If you don't mind a somewhat bonkers idea: how about a JS client-side plugin capability that allows any user looking at a Datasette site to pull in external plugins for data manipulation, even if the Datasette owner hasn't added them? (Yes, this may be much too ambitious. If you're remotely interested, maybe fork this discussion to a different issue.)

This is some fascinating reading about what JS sandboxing looks like these days: https://www.figma.com/blog/how-we-built-the-figma-plugin-system/

Codecov Report

Merging #1008 into main will increase coverage by 0.00%. The diff coverage is 100.00%.

```diff @@ Coverage Diff @@

main #1008 +/-

======================================= Coverage 84.55% 84.56%
======================================= Files 28 28
Lines 3878 3880 +2
======================================= + Hits 3279 3281 +2
Misses 599 599
```

| Impacted Files | Coverage Δ | | |---|---|---| | datasette/app.py | 96.35% <100.00%> (+0.01%) | :arrow_up: |

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1bdbc8a...4085898. Read the comment docs.

I can start by combining the release notes for the 0.50 alphas.

Documentation (from #1006): https://docs.datasette.io/en/latest/internals.html#client

https://docs.datasette.io/en/latest/internals.html#client

I agree on the descriptive and python-consistent naming. There is already a tojson, but frankly i find the "to" and "from" confusing in a text templating language where what's a string and what's data isn't 100% transparent.

I think json_dumps() and json_loads() as aliases for json.dumps() and json.loads() is the way to go here.

I don't appear to be using these anywhere, not sure why I spotted a warning (which I now can't find).

I'm not a huge fan of from_json as the name for this. Some other options:

• Expose json directly so templates can do json.loads() and json.dumps() - this allows for outputting JSON too, which is useful. But is there anything else on the json module that shouldn't be exposed in templates?
• json_dumps() and json_loads() template functions. I quite like that.
• Something else?

Yes I think that makes sense. I added json to the template context in Dogsheep Beta just a few days ago because I needed that: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/init.py#L176

This can become a section on https://docs.datasette.io/en/stable/internals.html

I'm going to document this in a separate issue.

Codecov Report

Merging #1000 into main will increase coverage by 0.15%. The diff coverage is 100.00%.

```diff @@ Coverage Diff @@

main #1000 +/-

========================================== + Coverage 84.37% 84.52% +0.15%
========================================== Files 28 28
Lines 3871 3878 +7
========================================== + Hits 3266 3278 +12
+ Misses 605 600 -5
```

| Impacted Files | Coverage Δ | | |---|---|---| | datasette/app.py | 96.34% <100.00%> (+0.02%) | :arrow_up: | | datasette/cli.py | 74.35% <100.00%> (ø) | | | datasette/utils/testing.py | 95.16% <100.00%> (-4.84%) | :arrow_down: | | datasette/views/base.py | 93.94% <100.00%> (+0.11%) | :arrow_up: | | datasette/utils/asgi.py | 91.92% <0.00%> (ø) | | | datasette/views/special.py | 93.51% <0.00%> (+8.33%) | :arrow_up: |

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7249ac5...8a80c79. Read the comment docs.

My httpx pull request adding raw_path support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.

I'm going to mark these tests as xfail so I can land this change - I'll remove that once an httpx release comes out that I can use to get the tests passing.

Here's where httpx sets up the ASGI scope: https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L82-L97

python # ASGI scope. scheme, host, port, full_path = url path, _, query = full_path.partition(b"?") scope = { "type": "http", "asgi": {"version": "3.0"}, "http_version": "1.1", "method": method.decode(), "headers": [(k.lower(), v) for (k, v) in headers], "scheme": scheme.decode("ascii"), "path": unquote(path.decode("ascii")), "query_string": query, "server": (host.decode("ascii"), port), "client": self.client, "root_path": self.root_path, } Sure enough, it doesn't set the raw_path.

I may need to fuss around with how the httpx client sends things to the ASGI app.

https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L26 is relevant:

Alternatively, you can setup the transport instance explicitly.
This allows you to include any additional configuration arguments specific
to the ASGITransport class:
```
transport = httpx.ASGITransport(
    app=app,
    root_path="/submount",
    client=("1.2.3.4", 123)
)
client = httpx.AsyncClient(transport=transport)
```

I'm testing this with a print(scope) and pytest -k test_table_with_slashes_in_name -s.

Against the main branch:

{'type': 'http', 'http_version': '1.0', 'method': 'GET', 'path': '/fixtures/table/with/slashes.csv', 'raw_path': b'/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'headers': [[b'host', b'localhost']], 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x10e2e6040>}

Against the broken branch:

tests/test_api.py {'type': 'http', 'asgi': {'version': '3.0'}, 'http_version': '1.1', 'method': 'GET', 'headers': [(b'host', b'localhost'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'user-agent', b'python-httpx/0.15.0')], 'scheme': 'http', 'path': '/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'server': ('localhost', None), 'client': ('127.0.0.1', 123), 'root_path': '', 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x109e0eca0>}

This is on my laptop though so both of those pass the tests.

Key difference: the httpx version doesn't set a raw_path at all. BUT.. it does set path and sets it to '/fixtures/table%2Fwith%2Fslashes.csv'

The non-httpx version sets raw_path to b'/fixtures/table%2Fwith%2Fslashes.csv' and path to '/fixtures/table/with/slashes.csv'.

Most likely reason for those failures is that path and raw_path are not being simulated correctly. I used to do those here:

https://github.com/simonw/datasette/blob/402cf870b7d65f9b5fba9e23aa99433294bd4523/datasette/utils/testing.py#L116-L125

But now I'm delegating that to httpx to handle.

WEIRD that it passes on my laptop but fails in GitHub Actions CI though.

This is really weird: new set of test failures that I wasn't seeing before, and those tests aren't failing on my laptop: =========================== short test summary info ============================ FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200 FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200 FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert... FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5] FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] ================== 5 failed, 738 passed in 194.73s (0:03:14) ===================

These failures are giving me a severe "how did this ever work in the first place?" vibe: FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] I have a fix for them, no idea why they weren't already failing though.

FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages'

That one is caused by response.cookies skipping cookies that were set to the empty string. Same fix as this: https://github.com/simonw/datasette/blob/a1687351fb75b01f737fda4ad07e0781029de05c/tests/test_auth.py#L90-L95

Still need to handle these six failing tests: FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'elemen... FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', '... FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<... FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<p cla... FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages' FAILED tests/test_plugins.py::test_hook_register_magic_parameters - AssertionError: assert [{'line': '1.0', 'rowid': 1}] == [{'line': '1.1', 'rowid': 1}] =========================================================== 6 failed, 731 passed, 6 warnings in 129.17s (0:02:09) ===========================================================

With the single client that is reused for all tests: % time pytest tests/test_api.py ... 6.73s user 9.91s system 81% cpu 20.365 total After switching back to this class: ```python class DatasetteClient: def init(self, ds): self.app = ds.app()

def _fix(self, path):
    if path.startswith("/"):
        path = "http://localhost{}".format(path)
    return path

async def get(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.get(self._fix(path), **kwargs)

async def options(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.options(self._fix(path), **kwargs)

async def head(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.head(self._fix(path), **kwargs)

async def post(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.post(self._fix(path), **kwargs)

async def put(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.put(self._fix(path), **kwargs)

async def patch(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.patch(self._fix(path), **kwargs)

async def delete(self, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.delete(self._fix(path), **kwargs)

async def request(self, method, path, **kwargs):
    async with httpx.AsyncClient(app=self.app) as client:
        return await client.request(method, self._fix(path), **kwargs)

The time taken is: % time pytest tests/test_api.py ... 7.26s user 10.02s system 82% cpu 21.014 total ``` That's close enough that I don't feel I need to investigate this further.

I'm going to switch back to having each request run through a new client. I'm worried about the impact on test performance though. I'll run a microbenchmark before and after.

The topic of disabling cookie persistence is discussed a little here: https://github.com/encode/httpx/issues/422#issuecomment-537906693

We have a cookiejar abstraction, I think setting it to an always-empty jar like you describe is best. :)

For this failing test I'm suspicious that the AsyncClient may be persisting cookies in between requests: ``` def test_actor_cookie(app_client): "A valid actor cookie sets request.scope['actor']" cookie = app_client.actor_cookie({"id": "test"}) response = app_client.get("/", cookies={"ds_actor": cookie})

  assert {"id": "test"} == app_client.ds._last_request.scope["actor"]

E AssertionError: assert {'id': 'test'} == {'id': 'root'} E Differing items: E {'id': 'test'} != {'id': 'root'} E Use -v to get the full diff ```

I actually have a sensible OPTIONS implementation here:

https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/datasette/views/base.py#L154-L165

I'm going to set the default one to return a 405 (Method Not Allowed) like Google does.

I'm tempted to imitate Django Rest Framework here: https://github.com/encode/django-rest-framework/blob/2e721cdbc85a924d0b0f093b86fe1497b58fe287/rest_framework/views.py#L514-L521 python def options(self, request, *args, **kwargs): """ Handler method for HTTP 'OPTIONS' request. """ if self.metadata_class is None: return self.http_method_not_allowed(request, *args, **kwargs) data = self.metadata_class().determine_metadata(request, self) return Response(data, status=status.HTTP_200_OK) That determine_metadata() default method does this: https://github.com/encode/django-rest-framework/blob/335054a5d36b352a58286b303b608b6bf48152f8/rest_framework/metadata.py#L29

Note the comment at the top: There are not any formalized standards for `OPTIONS` responses for us to base this on.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS

The www.djangoproject.com site returns an empty page: ``` ~ % curl -vv -XOPTIONS https://www.djangoproject.com/ * Trying 151.101.42.217:443... * Connected to www.djangoproject.com (151.101.42.217) port 443 (#0) * ALPN, offering http/1.1 * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: osff2.map.fastly.net * Server certificate: GlobalSign CloudSSL CA - SHA256 - G3 * Server certificate: GlobalSign Root CA

OPTIONS / HTTP/1.1 Host: www.djangoproject.com User-Agent: curl/7.70.0 Accept: /

• Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Connection: keep-alive < Content-Length: 0 < Server: nginx < Content-Type: text/html; charset=utf-8 < Allow: GET, HEAD, OPTIONS < Content-Language: en < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < X-XSS-Protection: 1; mode=block < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload < Access-Control-Allow-Origin: https://code.djangoproject.com < Accept-Ranges: bytes < Date: Fri, 09 Oct 2020 00:59:42 GMT < Via: 1.1 varnish < X-Served-By: cache-sjc10047-SJC < X-Cache: MISS < X-Cache-Hits: 0 < X-Timer: S1602205182.833493,VS0,VE305 < Vary: Accept-Language, Accept-Encoding <
• Connection #0 to host www.djangoproject.com left intact ~ % ```

``` ~ % curl -XOPTIONS https://www.google.com/

What should an OPTIONS request return, anyway?

So the bug is in this code here: https://github.com/simonw/datasette/blob/703439bdc37e724b01bc6d7a1fc1d955795132f2/datasette/views/base.py#L113-L115

To get a traceback: datasette . -p 8009 --pdb And then: curl -XOPTIONS http://127.0.0.1:8009 This causes the server to open a debugging prompt: ``` INFO: 127.0.0.1:59514 - "OPTIONS / HTTP/1.1" 500 Internal Server Error

/Users/simon/Dropbox/Development/datasette/datasette/views/base.py(115)dispatch_request() -> return await handler(request, args, kwargs) (Pdb) list 110 def database_color(self, database): 111 return "ff0000" 112
113 async def dispatch_request(self, request, args, kwargs): 114 handler = getattr(self, request.method.lower(), None) 115 -> return await handler(request, *args, kwargs) 116
117 async def render(self, templates, request, context=None): 118 context = context or {} 119 template = self.ds.jinja_env.select_template(templates) 120 template_context = { (Pdb) ```

Almost all of the tests are passing: =========================== short test summary info ============================ FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200 FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200 FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert... FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5] FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyE... FAILED tests/test_plugins.py::test_hook_register_magic_parameters - Assertion... ============ 11 failed, 718 passed, 6 warnings in 225.77s (0:03:45) ============

I'm going to route the existing TestClient through this mechanism to exercise it during the tests.