{"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-706269271", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 706269271, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjI2OTI3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T16:08:49Z", "updated_at": "2020-10-09T16:08:49Z", "author_association": "OWNER", "body": "I'm going to document this in a separate issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705890365", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705890365, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTg5MDM2NQ==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2020-10-09T00:03:29Z", "updated_at": "2020-10-09T16:07:03Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=h1) Report\n> Merging [#1000](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/7249ac5ca04b5ddc6517750326ee7e522cc49145?el=desc) will **increase** coverage by `0.15%`.\n> The diff coverage is `100.00%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1000/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=tree)\n\n```diff\n@@ Coverage Diff @@\n## main #1000 +/- ##\n==========================================\n+ Coverage 84.37% 84.52% +0.15% \n==========================================\n Files 28 28 \n Lines 3871 3878 +7 \n==========================================\n+ Hits 3266 3278 +12 \n+ Misses 605 600 -5 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=tree) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.34% <100.00%> (+0.02%)` | :arrow_up: |\n| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `74.35% <100.00%> (\u00f8)` | |\n| [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.16% <100.00%> (-4.84%)` | :arrow_down: |\n| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.94% <100.00%> (+0.11%)` | :arrow_up: |\n| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.92% <0.00%> (\u00f8)` | |\n| [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `93.51% <0.00%> (+8.33%)` | :arrow_up: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=footer). Last update [7249ac5...8a80c79](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-706263157", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 706263157, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjI2MzE1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T15:57:15Z", "updated_at": "2020-10-09T15:57:15Z", "author_association": "OWNER", "body": "My `httpx` pull request adding `raw_path` support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.\r\n\r\nI'm going to mark these tests as `xfail` so I can land this change - I'll remove that once an `httpx` release comes out that I can use to get the tests passing.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705946360", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705946360, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTk0NjM2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T03:28:08Z", "updated_at": "2020-10-09T03:28:08Z", "author_association": "OWNER", "body": "Here's where `httpx` sets up the ASGI scope: https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L82-L97\r\n\r\n```python\r\n # ASGI scope.\r\n scheme, host, port, full_path = url\r\n path, _, query = full_path.partition(b\"?\")\r\n scope = {\r\n \"type\": \"http\",\r\n \"asgi\": {\"version\": \"3.0\"},\r\n \"http_version\": \"1.1\",\r\n \"method\": method.decode(),\r\n \"headers\": [(k.lower(), v) for (k, v) in headers],\r\n \"scheme\": scheme.decode(\"ascii\"),\r\n \"path\": unquote(path.decode(\"ascii\")),\r\n \"query_string\": query,\r\n \"server\": (host.decode(\"ascii\"), port),\r\n \"client\": self.client,\r\n \"root_path\": self.root_path,\r\n }\r\n```\r\nSure enough, it doesn't set the `raw_path`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705946120", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705946120, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTk0NjEyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T03:27:05Z", "updated_at": "2020-10-09T03:27:05Z", "author_association": "OWNER", "body": "I may need to fuss around with how the `httpx` client sends things to the ASGI app.\r\n\r\nhttps://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L26 is relevant:\r\n\r\n Alternatively, you can setup the transport instance explicitly.\r\n This allows you to include any additional configuration arguments specific\r\n to the ASGITransport class:\r\n ```\r\n transport = httpx.ASGITransport(\r\n app=app,\r\n root_path=\"/submount\",\r\n client=(\"1.2.3.4\", 123)\r\n )\r\n client = httpx.AsyncClient(transport=transport)\r\n ```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705945591", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705945591, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTk0NTU5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T03:24:48Z", "updated_at": "2020-10-09T03:24:48Z", "author_association": "OWNER", "body": "I'm testing this with a `print(scope)` and `pytest -k test_table_with_slashes_in_name -s`.\r\n\r\nAgainst the `main` branch:\r\n\r\n`{'type': 'http', 'http_version': '1.0', 'method': 'GET', 'path': '/fixtures/table/with/slashes.csv', 'raw_path': b'/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'headers': [[b'host', b'localhost']], 'csrftoken': ._asgi_csrf_decorator..app_wrapped_with_csrf..get_csrftoken at 0x10e2e6040>}`\r\n\r\nAgainst the broken branch:\r\n\r\n`tests/test_api.py {'type': 'http', 'asgi': {'version': '3.0'}, 'http_version': '1.1', 'method': 'GET', 'headers': [(b'host', b'localhost'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'user-agent', b'python-httpx/0.15.0')], 'scheme': 'http', 'path': '/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'server': ('localhost', None), 'client': ('127.0.0.1', 123), 'root_path': '', 'csrftoken': ._asgi_csrf_decorator..app_wrapped_with_csrf..get_csrftoken at 0x109e0eca0>}`\r\n\r\nThis is on my laptop though so both of those pass the tests.\r\n\r\nKey difference: the `httpx` version doesn't set a `raw_path` at all. BUT.. it does set `path` and sets it to `'/fixtures/table%2Fwith%2Fslashes.csv'`\r\n\r\nThe non-httpx version sets `raw_path` to `b'/fixtures/table%2Fwith%2Fslashes.csv'` and `path` to `'/fixtures/table/with/slashes.csv'`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705941580", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705941580, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTk0MTU4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T03:08:43Z", "updated_at": "2020-10-09T03:08:43Z", "author_association": "OWNER", "body": "Most likely reason for those failures is that `path` and `raw_path` are not being simulated correctly. I used to do those here:\r\n\r\nhttps://github.com/simonw/datasette/blob/402cf870b7d65f9b5fba9e23aa99433294bd4523/datasette/utils/testing.py#L116-L125\r\n\r\nBut now I'm delegating that to `httpx` to handle.\r\n\r\nWEIRD that it passes on my laptop but fails in GitHub Actions CI though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705940507", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705940507, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTk0MDUwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T03:04:15Z", "updated_at": "2020-10-09T03:04:15Z", "author_association": "OWNER", "body": "This is really weird: new set of test failures that I wasn't seeing before, and those tests aren't failing on my laptop:\r\n```\r\n=========================== short test summary info ============================\r\nFAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200\r\nFAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200\r\nFAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert...\r\nFAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5]\r\nFAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html]\r\n================== 5 failed, 738 passed in 194.73s (0:03:14) ===================\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705937696", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705937696, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTkzNzY5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T02:52:53Z", "updated_at": "2020-10-09T02:52:53Z", "author_association": "OWNER", "body": "These failures are giving me a severe \"how did this ever work in the first place?\" vibe:\r\n```\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/]\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/]\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/]\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/]\r\n```\r\nI have a fix for them, no idea why they weren't already failing though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705926445", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705926445, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTkyNjQ0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T02:15:38Z", "updated_at": "2020-10-09T02:15:38Z", "author_association": "OWNER", "body": "> FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages'\r\n\r\nThat one is caused by `response.cookies` skipping cookies that were set to the empty string. Same fix as this: https://github.com/simonw/datasette/blob/a1687351fb75b01f737fda4ad07e0781029de05c/tests/test_auth.py#L90-L95", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705926035", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705926035, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTkyNjAzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T02:14:14Z", "updated_at": "2020-10-09T02:14:14Z", "author_association": "OWNER", "body": "Still need to handle these six failing tests:\r\n```\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'elemen...\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', '...\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<...\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '

We have a cookiejar abstraction, I think setting it to an always-empty jar like you describe is best. :)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705918844", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705918844, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTkxODg0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T01:46:06Z", "updated_at": "2020-10-09T01:46:06Z", "author_association": "OWNER", "body": "For this failing test I'm suspicious that the AsyncClient may be persisting cookies in between requests:\r\n```\r\n def test_actor_cookie(app_client):\r\n \"A valid actor cookie sets request.scope['actor']\"\r\n cookie = app_client.actor_cookie({\"id\": \"test\"})\r\n response = app_client.get(\"/\", cookies={\"ds_actor\": cookie})\r\n> assert {\"id\": \"test\"} == app_client.ds._last_request.scope[\"actor\"]\r\nE AssertionError: assert {'id': 'test'} == {'id': 'root'}\r\nE Differing items:\r\nE {'id': 'test'} != {'id': 'root'}\r\nE Use -v to get the full diff\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705902902", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705902902, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTkwMjkwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T00:50:49Z", "updated_at": "2020-10-09T00:50:49Z", "author_association": "OWNER", "body": "Almost all of the tests are passing:\r\n```\r\n=========================== short test summary info ============================\r\nFAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200\r\nFAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200\r\nFAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert...\r\nFAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5]\r\nFAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html]\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/]\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/]\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/]\r\nFAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/]\r\nFAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyE...\r\nFAILED tests/test_plugins.py::test_hook_register_magic_parameters - Assertion...\r\n============ 11 failed, 718 passed, 6 warnings in 225.77s (0:03:45) ============\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705899629", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705899629, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTg5OTYyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-09T00:37:02Z", "updated_at": "2020-10-09T00:37:02Z", "author_association": "OWNER", "body": "I'm going to route the existing `TestClient` through this mechanism to exercise it during the tests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1000#issuecomment-705889120", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1000", "id": 705889120, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNTg4OTEyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-08T23:59:01Z", "updated_at": "2020-10-08T23:59:01Z", "author_association": "OWNER", "body": "Needs tests and documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 717746043, "label": "datasette.client internal requests mechanism"}, "performed_via_github_app": null}