issue_comments

I can start by combining the release notes for the 0.50 alphas.

Documentation (from #1006): https://docs.datasette.io/en/latest/internals.html#client

https://docs.datasette.io/en/latest/internals.html#client

I think json_dumps() and json_loads() as aliases for json.dumps() and json.loads() is the way to go here.

I don't appear to be using these anywhere, not sure why I spotted a warning (which I now can't find).

I'm not a huge fan of from_json as the name for this. Some other options:

• Expose json directly so templates can do json.loads() and json.dumps() - this allows for outputting JSON too, which is useful. But is there anything else on the json module that shouldn't be exposed in templates?
• json_dumps() and json_loads() template functions. I quite like that.
• Something else?

Yes I think that makes sense. I added json to the template context in Dogsheep Beta just a few days ago because I needed that: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/init.py#L176

This can become a section on https://docs.datasette.io/en/stable/internals.html

I'm going to document this in a separate issue.

My httpx pull request adding raw_path support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.

I'm going to mark these tests as xfail so I can land this change - I'll remove that once an httpx release comes out that I can use to get the tests passing.

Here's where httpx sets up the ASGI scope: https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L82-L97

python # ASGI scope. scheme, host, port, full_path = url path, _, query = full_path.partition(b"?") scope = { "type": "http", "asgi": {"version": "3.0"}, "http_version": "1.1", "method": method.decode(), "headers": [(k.lower(), v) for (k, v) in headers], "scheme": scheme.decode("ascii"), "path": unquote(path.decode("ascii")), "query_string": query, "server": (host.decode("ascii"), port), "client": self.client, "root_path": self.root_path, } Sure enough, it doesn't set the raw_path.

I may need to fuss around with how the httpx client sends things to the ASGI app.

https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L26 is relevant:

Alternatively, you can setup the transport instance explicitly.
This allows you to include any additional configuration arguments specific
to the ASGITransport class:
```
transport = httpx.ASGITransport(
    app=app,
    root_path="/submount",
    client=("1.2.3.4", 123)
)
client = httpx.AsyncClient(transport=transport)
```

I'm testing this with a print(scope) and pytest -k test_table_with_slashes_in_name -s.

Against the main branch:

{'type': 'http', 'http_version': '1.0', 'method': 'GET', 'path': '/fixtures/table/with/slashes.csv', 'raw_path': b'/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'headers': [[b'host', b'localhost']], 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x10e2e6040>}

Against the broken branch:

tests/test_api.py {'type': 'http', 'asgi': {'version': '3.0'}, 'http_version': '1.1', 'method': 'GET', 'headers': [(b'host', b'localhost'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'user-agent', b'python-httpx/0.15.0')], 'scheme': 'http', 'path': '/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'server': ('localhost', None), 'client': ('127.0.0.1', 123), 'root_path': '', 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x109e0eca0>}

This is on my laptop though so both of those pass the tests.

Key difference: the httpx version doesn't set a raw_path at all. BUT.. it does set path and sets it to '/fixtures/table%2Fwith%2Fslashes.csv'

The non-httpx version sets raw_path to b'/fixtures/table%2Fwith%2Fslashes.csv' and path to '/fixtures/table/with/slashes.csv'.

Most likely reason for those failures is that path and raw_path are not being simulated correctly. I used to do those here:

https://github.com/simonw/datasette/blob/402cf870b7d65f9b5fba9e23aa99433294bd4523/datasette/utils/testing.py#L116-L125

But now I'm delegating that to httpx to handle.

WEIRD that it passes on my laptop but fails in GitHub Actions CI though.

This is really weird: new set of test failures that I wasn't seeing before, and those tests aren't failing on my laptop: =========================== short test summary info ============================ FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200 FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200 FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert... FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5] FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] ================== 5 failed, 738 passed in 194.73s (0:03:14) ===================

These failures are giving me a severe "how did this ever work in the first place?" vibe: FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] I have a fix for them, no idea why they weren't already failing though.

FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages'

That one is caused by response.cookies skipping cookies that were set to the empty string. Same fix as this: https://github.com/simonw/datasette/blob/a1687351fb75b01f737fda4ad07e0781029de05c/tests/test_auth.py#L90-L95

Still need to handle these six failing tests: FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'elemen... FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', '... FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<... FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<p cla... FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages' FAILED tests/test_plugins.py::test_hook_register_magic_parameters - AssertionError: assert [{'line': '1.0', 'rowid': 1}] == [{'line': '1.1', 'rowid': 1}] =========================================================== 6 failed, 731 passed, 6 warnings in 129.17s (0:02:09) ===========================================================

With the single client that is reused for all tests: % time pytest tests/test_api.py ... 6.73s user 9.91s system 81% cpu 20.365 total After switching back to this class: ```python class DatasetteClient: def init(self, ds): self.app = ds.app()

def _fix(self, path):
    if path.startswith("/"):
        path = "http://localhost{}".format(path)
    return path

async def get(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.get(self._fix(path), **kwargs)

async def options(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.options(self._fix(path), **kwargs)

async def head(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.head(self._fix(path), **kwargs)

async def post(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.post(self._fix(path), **kwargs)

async def put(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.put(self._fix(path), **kwargs)

async def patch(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.patch(self._fix(path), **kwargs)

async def delete(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.delete(self._fix(path), **kwargs)

async def request(self, method, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.request(method, self._fix(path), **kwargs)

The time taken is: % time pytest tests/test_api.py ... 7.26s user 10.02s system 82% cpu 21.014 total ``` That's close enough that I don't feel I need to investigate this further.

I'm going to switch back to having each request run through a new client. I'm worried about the impact on test performance though. I'll run a microbenchmark before and after.

The topic of disabling cookie persistence is discussed a little here: https://github.com/encode/httpx/issues/422#issuecomment-537906693

We have a cookiejar abstraction, I think setting it to an always-empty jar like you describe is best. :)

For this failing test I'm suspicious that the AsyncClient may be persisting cookies in between requests: ``` def test_actor_cookie(app_client): "A valid actor cookie sets request.scope['actor']" cookie = app_client.actor_cookie({"id": "test"}) response = app_client.get("/", cookies={"ds_actor": cookie})

  assert {"id": "test"} == app_client.ds._last_request.scope["actor"]

E AssertionError: assert {'id': 'test'} == {'id': 'root'} E Differing items: E {'id': 'test'} != {'id': 'root'} E Use -v to get the full diff ```

I actually have a sensible OPTIONS implementation here:

https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/datasette/views/base.py#L154-L165

I'm going to set the default one to return a 405 (Method Not Allowed) like Google does.

I'm tempted to imitate Django Rest Framework here: https://github.com/encode/django-rest-framework/blob/2e721cdbc85a924d0b0f093b86fe1497b58fe287/rest_framework/views.py#L514-L521 python def options(self, request, *args, **kwargs): """ Handler method for HTTP 'OPTIONS' request. """ if self.metadata_class is None: return self.http_method_not_allowed(request, *args, **kwargs) data = self.metadata_class().determine_metadata(request, self) return Response(data, status=status.HTTP_200_OK) That determine_metadata() default method does this: https://github.com/encode/django-rest-framework/blob/335054a5d36b352a58286b303b608b6bf48152f8/rest_framework/metadata.py#L29

Note the comment at the top: There are not any formalized standards for `OPTIONS` responses for us to base this on.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS

The www.djangoproject.com site returns an empty page: ``` ~ % curl -vv -XOPTIONS https://www.djangoproject.com/ * Trying 151.101.42.217:443... * Connected to www.djangoproject.com (151.101.42.217) port 443 (#0) * ALPN, offering http/1.1 * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: osff2.map.fastly.net * Server certificate: GlobalSign CloudSSL CA - SHA256 - G3 * Server certificate: GlobalSign Root CA

OPTIONS / HTTP/1.1 Host: www.djangoproject.com User-Agent: curl/7.70.0 Accept: /

• Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Connection: keep-alive < Content-Length: 0 < Server: nginx < Content-Type: text/html; charset=utf-8 < Allow: GET, HEAD, OPTIONS < Content-Language: en < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < X-XSS-Protection: 1; mode=block < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload < Access-Control-Allow-Origin: https://code.djangoproject.com < Accept-Ranges: bytes < Date: Fri, 09 Oct 2020 00:59:42 GMT < Via: 1.1 varnish < X-Served-By: cache-sjc10047-SJC < X-Cache: MISS < X-Cache-Hits: 0 < X-Timer: S1602205182.833493,VS0,VE305 < Vary: Accept-Language, Accept-Encoding <
• Connection #0 to host www.djangoproject.com left intact ~ % ```

``` ~ % curl -XOPTIONS https://www.google.com/

What should an OPTIONS request return, anyway?

So the bug is in this code here: https://github.com/simonw/datasette/blob/703439bdc37e724b01bc6d7a1fc1d955795132f2/datasette/views/base.py#L113-L115

To get a traceback: datasette . -p 8009 --pdb And then: curl -XOPTIONS http://127.0.0.1:8009 This causes the server to open a debugging prompt: ``` INFO: 127.0.0.1:59514 - "OPTIONS / HTTP/1.1" 500 Internal Server Error

/Users/simon/Dropbox/Development/datasette/datasette/views/base.py(115)dispatch_request() -> return await handler(request, args, kwargs) (Pdb) list 110 def database_color(self, database): 111 return "ff0000" 112
113 async def dispatch_request(self, request, args, kwargs): 114 handler = getattr(self, request.method.lower(), None) 115 -> return await handler(request, *args, kwargs) 116
117 async def render(self, templates, request, context=None): 118 context = context or {} 119 template = self.ds.jinja_env.select_template(templates) 120 template_context = { (Pdb) ```

Almost all of the tests are passing: =========================== short test summary info ============================ FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200 FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200 FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert... FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5] FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyE... FAILED tests/test_plugins.py::test_hook_register_magic_parameters - Assertion... ============ 11 failed, 718 passed, 6 warnings in 225.77s (0:03:45) ============

I'm going to route the existing TestClient through this mechanism to exercise it during the tests.