issue_comments

Maybe instead of implementing datasette.get() and datasette.post() and datasette.request() and datasette.stream() I could instead have a nested object called datasette.client which is a preconfigured AsyncClient instance.

python response = await datasette.client.get("/") Or perhaps this should be a method in case I ever need to be able to await it: python response = await (await datasette.client()).get("/") This is a bit cosmetically ugly though, I'd rather avoid that if possible.

Maybe I could get this working by returning an object from .client() which provides a await obj.get() method: python response = await datasette.client().get("/") I don't think there's any benefit to that over await datasette.client.get() though.

Should I instantiate a single Client and reuse it for all internal requests, or can I instantiate a new Client for each request?

https://www.python-httpx.org/advanced/#why-use-a-client says that the main benefit of a Client instance is HTTP connection pooling - which isn't an issue for these internal requests since they won't be using the HTTP protocol at all, they'll be calling the ASGI application directly.

So I'm leaning towards instantiating a fresh client for every internal request. I'll run a microbenchmark to check that this doesn't have any unpleasant performance implications.

I think I can use async with httpx.AsyncClient(base_url="http://localhost/") as client: to ensure I don't need to use http://localhost/ on every call.

I could solve streaming using something like this: python async with datasette.stream("GET", "/fixtures/compound_three_primary_keys.csv?_stream=on&_size=max") as response: async for chunk in response.aiter_bytes(): print(chunk) Which would be a wrapper around AsyncClient.stream(method, url, ...) from https://www.python-httpx.org/async/#streaming-responses

What if datasette.get() was an alias for httpx.get(), pre-configured to route to the correct application? And with some sugar that added http://localhost/ to the beginning of the path if it was missing?

This would make httpx a dependency of core Datasette, which I think is OK.

It would also solve the return type problem: I would return whatever httpx returns.

I'm tempted to create a await datasette.request() method which can take any HTTP verb - then have datasette.get() and datasette.post() as thin wrappers around it.

Maybe these methods become the way most Datasette tests are written, replacing the existing TestClient mechanism?

So what should I do about streaming responses?

I could deliberately ignore them - through an exception if you attempt to run await datasette.get(...) against a streaming URL.

I could load the entire response into memory and return it as a wrapped object.

I could support some kind of asynchronous iterator mechanism. This would be pretty elegant if I could decide the right syntax for it - it would allow plugins to take advantage of other internal URLs that return streaming content without needing to load that content entirely into memory in order to process it.

I'm not going to mess around with formats - you'll get back the exact response that a web client would receive.

Question: what should the response object look like? e.g. if you do:

response = await datasette.get("/db/table.json")

What should response be?

I could reuse the Datasette Response class from datasette.utils.asgi. This would work well for regular responses which just have a status code, some headers and a response body. It wouldn't be great for streaming responses though such as you get back from ?_stream=1 CSV exports.

Here's what that looks like: Traceback (most recent call last): File "/Users/simon/Dropbox/Development/datasette/plugins/sql_error.py", line 5, in oh_no_error return 100 / 0 ZeroDivisionError: division by zero ERROR: conn=<sqlite3.Connection object at 0x10bce0030>, sql = 'select oh_no_error()', params = {}: user-defined function raised exception INFO: 127.0.0.1:54066 - "GET /data?sql=select+oh_no_error%28%29 HTTP/1.1" 400 Bad Request

I'm going to turn this on. If people complain about it I can turn it off again (or make it a configuration setting).

Here's the commit (from 15 years ago) where enable_callback_tracebacks was first added: https://github.com/ghaering/pysqlite/commit/1e8bd36be93b7d7425910642b72e4152c77b0dfd

• Exceptions in callbacks lead to the query being aborted now instead of silently leading to generating values.
• Exceptions in callbacks can be echoed to stderr if you call the module level function enable_callback_tracebacks: enable_callback_tracebacks(1).

I've been trying to figure out why this is an optional setting that defaults to off.

I think it's because it writes directly to stderr, so the maintainers of sqlite3 reasonably decided that people should be able to opt in to that rather than having weird stuff show up on stderr that they weren't expecting.

https://github.com/peter-wangxu/persist-queue/issues/74 warns that this might not work with PyPy.

I could solve that with: python if hasattr(sqlite3, "enable_callback_tracebacks"): sqlite3.enable_callback_tracebacks(True)

I'm not going to drop CSRF protection - it's still needed for older browsers - but I have relaxed the circumstances under which it is applied. It only applies to requests that include cookies for example, so API clients that don't send cookies don't need to worry about it.

Thanks - I've fixed this in datasette-media and the other plugins that use that hook now I think.

Thanks, I've fixed that now. It only affected the GitHub release notes - the ones at https://docs.datasette.io/en/stable/changelog.html#v0-45 had the correct links.

I fixed this in 5e0b72247ecab4ce0fcec599b77a83d73a480872

I've been running on main for a while now with no issues.

https://hub.docker.com/r/datasetteproject/datasette/tags - 0.49.1 was successfully pushed to Docker Hub by https://github.com/simonw/datasette/runs/1119815175?check_suite_focus=true

I think I've fixed this with recent changes I made as part of #941 - but I won't know until I release the next version.

Can't reproduce this bug now.

I fixed this in ea340cf320a2566d24517fb4a0c9852c5059e771 for #963 (a duplicate of this issue).

I think the easiest fix is for me to ensure that calls to __len__ on the MagicParameters class always return at least 1.

I wish I could call https://www.sqlite.org/c3ref/bind_parameter_count.html and https://www.sqlite.org/c3ref/bind_parameter_name.html from Python.

Might be possible to do that using ctypes - see this example code: https://mail.python.org/pipermail//pypy-commit/2013-February/071372.html

python param_count = lib.sqlite3_bind_parameter_count(self.statement) for idx in range(1, param_count + 1): param_name = lib.sqlite3_bind_parameter_name(self.statement, idx)

So the problem actually occurs when the MagicParameters class wraps an empty dictionary.

Relevant code:

https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L228-L236

And:

https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L364-L383

I'm passing a special magic parameters dictionary for the Python sqlite3 module to look up parameters in. When that dictionary is {} a __len__ check is performed on that dictionary, the result comes back as 0 and as a result it assumes there are no parameters.

I tracked down the relevant C code:

https://github.com/python/cpython/blob/81715808716198471fbca0a3db42ac408468dbc5/Modules/_sqlite/statement.c#L218-L237

```c Py_BEGIN_ALLOW_THREADS num_params_needed = sqlite3_bind_parameter_count(self->st); Py_END_ALLOW_THREADS

if (PyTuple_CheckExact(parameters) || PyList_CheckExact(parameters) || (!PyDict_Check(parameters) && PySequence_Check(parameters))) {
    /* parameters passed as sequence */
    if (PyTuple_CheckExact(parameters)) {
        num_params = PyTuple_GET_SIZE(parameters);
    } else if (PyList_CheckExact(parameters)) {
        num_params = PyList_GET_SIZE(parameters);
    } else {
        num_params = PySequence_Size(parameters);
    }
    if (num_params != num_params_needed) {
        PyErr_Format(pysqlite_ProgrammingError,
                     "Incorrect number of bindings supplied. The current "
                     "statement uses %d, and there are %zd supplied.",
                     num_params_needed, num_params);
        return;
    }

```

It looks to me like this should fail if the number of keys known to be in the dictionary differs from the number of named parameters in the query. But if those numbers fail to match it still works as far as I can tell - it's only dictionary length of 0 that is causing the problems.

Yes! The tests all pass if I update the test function to do this: python response = magic_parameters_client.post( "/data/runme_post{}".format(qs), {"ignore_me": "1"}, csrftoken_from=use_csrf or None, allow_redirects=False, ) So the bug only occurs if the POST body is completely empty.

While I'm running the above test, in the rounds that work the receive() awaitable returns {'type': 'http.request', 'body': b'csrftoken=IlpwUGlSMFVVa3Z3ZlVoamQi.uY2U1tF4i0M-5M6x34vnBCmJgr0'}

In the rounds that fails it returns {'type': 'http.request'}

So it looks like the csrftoken_from=True parameter may be helping just by ensuring the body key is present and not missing. I wonder if it would work if a body of b'' was present there?

New (failing) test: python @pytest.mark.parametrize("use_csrf", [True, False]) @pytest.mark.parametrize("return_json", [True, False]) def test_magic_parameters_csrf_json(magic_parameters_client, use_csrf, return_json): magic_parameters_client.ds._metadata["databases"]["data"]["queries"]["runme_post"][ "sql" ] = "insert into logs (line) values (:_header_host)" qs = "" if return_json: qs = "?_json=1" response = magic_parameters_client.post( "/data/runme_post{}".format(qs), {}, csrftoken_from=use_csrf or None, allow_redirects=False, ) if return_json: assert response.status == 200 assert response.json["ok"], response.json else: assert response.status == 302 messages = magic_parameters_client.ds.unsign( response.cookies["ds_messages"], "messages" ) assert [["Query executed, 1 row affected", 1]] == messages post_actual = magic_parameters_client.get( "/data/logs.json?_sort_desc=rowid&_shape=array" ).json[0]["line"] assert post_actual == "localhost" It passes twice, fails twice - failures are for the ones where use_csrf is False.

Hunch: I think the asgi-csrf middleware may be consuming the request body and failing to restore it.

Is the bug here that magic parameters are incompatible with CSRF-exempt requests (e.g. request with no cookies)?

Relevant code: https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L222-L236

This issue may not be about _json=1 interacting with magic parameters after all.

So the mystery here is why does omitting csrftoken_from=True break the MagicParameters mechanism?

This is so weird. In the test I wrote for this the following passed:

response = magic_parameters_client.post("/data/runme_post?_json=1", {}, csrftoken_from=True)

But without the csrftoken_from=True parameter it failed with the bindings error:

response = magic_parameters_client.post("/data/runme_post?_json=1", {})

Here's the test I wrote:

python def test_magic_parameters_json_body(magic_parameters_client): magic_parameters_client.ds._metadata["databases"]["data"]["queries"]["runme_post"][ "sql" ] = "insert into logs (line) values (:_header_host)" response = magic_parameters_client.post("/data/runme_post?_json=1", {}, csrftoken_from=True) assert response.status == 200 assert response.json["ok"], response.json post_actual = magic_parameters_client.get( "/data/logs.json?_sort_desc=rowid&_shape=array" ).json[0]["line"]