issue_comments

11 rows where issue = 702069429 sorted by updated_at descending

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

user

issue

  • Writable canned queries with magic parameters fail if POST body is empty · 11

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
692951144 https://github.com/simonw/datasette/issues/967#issuecomment-692951144 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5Mjk1MTE0NA== simonw 9599 2020-09-15T20:08:12Z 2020-09-15T20:08:12Z OWNER

I think the easiest fix is for me to ensure that calls to __len__ on the MagicParameters class always return at least 1.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  
692946616 https://github.com/simonw/datasette/issues/967#issuecomment-692946616 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5Mjk0NjYxNg== simonw 9599 2020-09-15T19:59:21Z 2020-09-15T19:59:21Z OWNER

I wish I could call https://www.sqlite.org/c3ref/bind_parameter_count.html and https://www.sqlite.org/c3ref/bind_parameter_name.html from Python.

Might be possible to do that using ctypes - see this example code: https://mail.python.org/pipermail//pypy-commit/2013-February/071372.html

            param_count = lib.sqlite3_bind_parameter_count(self.statement)
            for idx in range(1, param_count + 1):
                param_name = lib.sqlite3_bind_parameter_name(self.statement,
                                                             idx)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  
692945504 https://github.com/simonw/datasette/issues/967#issuecomment-692945504 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5Mjk0NTUwNA== simonw 9599 2020-09-15T19:57:10Z 2020-09-15T19:57:10Z OWNER

So the problem actually occurs when the MagicParameters class wraps an empty dictionary.

Relevant code:

https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L228-L236

And:

https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L364-L383

I'm passing a special magic parameters dictionary for the Python sqlite3 module to look up parameters in. When that dictionary is {} a __len__ check is performed on that dictionary, the result comes back as 0 and as a result it assumes there are no parameters.

I tracked down the relevant C code:

https://github.com/python/cpython/blob/81715808716198471fbca0a3db42ac408468dbc5/Modules/_sqlite/statement.c#L218-L237

    Py_BEGIN_ALLOW_THREADS
    num_params_needed = sqlite3_bind_parameter_count(self->st);
    Py_END_ALLOW_THREADS

    if (PyTuple_CheckExact(parameters) || PyList_CheckExact(parameters) || (!PyDict_Check(parameters) && PySequence_Check(parameters))) {
        /* parameters passed as sequence */
        if (PyTuple_CheckExact(parameters)) {
            num_params = PyTuple_GET_SIZE(parameters);
        } else if (PyList_CheckExact(parameters)) {
            num_params = PyList_GET_SIZE(parameters);
        } else {
            num_params = PySequence_Size(parameters);
        }
        if (num_params != num_params_needed) {
            PyErr_Format(pysqlite_ProgrammingError,
                         "Incorrect number of bindings supplied. The current "
                         "statement uses %d, and there are %zd supplied.",
                         num_params_needed, num_params);
            return;
        }

It looks to me like this should fail if the number of keys known to be in the dictionary differs from the number of named parameters in the query. But if those numbers fail to match it still works as far as I can tell - it's only dictionary length of 0 that is causing the problems.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  
692940375 https://github.com/simonw/datasette/issues/967#issuecomment-692940375 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5Mjk0MDM3NQ== simonw 9599 2020-09-15T19:47:09Z 2020-09-15T19:47:09Z OWNER

Yes! The tests all pass if I update the test function to do this:

    response = magic_parameters_client.post(
        "/data/runme_post{}".format(qs),
        {"ignore_me": "1"},
        csrftoken_from=use_csrf or None,
        allow_redirects=False,
    )

So the bug only occurs if the POST body is completely empty.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  
692938935 https://github.com/simonw/datasette/issues/967#issuecomment-692938935 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5MjkzODkzNQ== simonw 9599 2020-09-15T19:44:21Z 2020-09-15T19:44:41Z OWNER

While I'm running the above test, in the rounds that work the receive() awaitable returns {'type': 'http.request', 'body': b'csrftoken=IlpwUGlSMFVVa3Z3ZlVoamQi.uY2U1tF4i0M-5M6x34vnBCmJgr0'}

In the rounds that fails it returns {'type': 'http.request'}

So it looks like the csrftoken_from=True parameter may be helping just by ensuring the body key is present and not missing. I wonder if it would work if a body of b'' was present there?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  
692937150 https://github.com/simonw/datasette/issues/967#issuecomment-692937150 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5MjkzNzE1MA== simonw 9599 2020-09-15T19:42:57Z 2020-09-15T19:42:57Z OWNER

New (failing) test:

@pytest.mark.parametrize("use_csrf", [True, False])
@pytest.mark.parametrize("return_json", [True, False])
def test_magic_parameters_csrf_json(magic_parameters_client, use_csrf, return_json):
    magic_parameters_client.ds._metadata["databases"]["data"]["queries"]["runme_post"][
        "sql"
    ] = "insert into logs (line) values (:_header_host)"
    qs = ""
    if return_json:
        qs = "?_json=1"
    response = magic_parameters_client.post(
        "/data/runme_post{}".format(qs),
        {},
        csrftoken_from=use_csrf or None,
        allow_redirects=False,
    )
    if return_json:
        assert response.status == 200
        assert response.json["ok"], response.json
    else:
        assert response.status == 302
        messages = magic_parameters_client.ds.unsign(
            response.cookies["ds_messages"], "messages"
        )
        assert [["Query executed, 1 row affected", 1]] == messages
    post_actual = magic_parameters_client.get(
        "/data/logs.json?_sort_desc=rowid&_shape=array"
    ).json[0]["line"]
    assert post_actual == "localhost"

It passes twice, fails twice - failures are for the ones where use_csrf is False.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  
692927867 https://github.com/simonw/datasette/issues/967#issuecomment-692927867 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5MjkyNzg2Nw== simonw 9599 2020-09-15T19:25:23Z 2020-09-15T19:25:23Z OWNER

Hunch: I think the asgi-csrf middleware may be consuming the request body and failing to restore it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  
692835066 https://github.com/simonw/datasette/issues/967#issuecomment-692835066 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5MjgzNTA2Ng== simonw 9599 2020-09-15T16:40:12Z 2020-09-15T16:40:12Z OWNER

Is the bug here that magic parameters are incompatible with CSRF-exempt requests (e.g. request with no cookies)?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  
692834670 https://github.com/simonw/datasette/issues/967#issuecomment-692834670 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5MjgzNDY3MA== simonw 9599 2020-09-15T16:39:29Z 2020-09-15T16:39:29Z OWNER

Relevant code: https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L222-L236

This issue may not be about _json=1 interacting with magic parameters after all.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  
692834064 https://github.com/simonw/datasette/issues/967#issuecomment-692834064 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5MjgzNDA2NA== simonw 9599 2020-09-15T16:38:21Z 2020-09-15T16:38:21Z OWNER

So the mystery here is why does omitting csrftoken_from=True break the MagicParameters mechanism?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  
692832113 https://github.com/simonw/datasette/issues/967#issuecomment-692832113 https://api.github.com/repos/simonw/datasette/issues/967 MDEyOklzc3VlQ29tbWVudDY5MjgzMjExMw== simonw 9599 2020-09-15T16:34:53Z 2020-09-15T16:37:43Z OWNER

This is so weird. In the test I wrote for this the following passed:

response = magic_parameters_client.post("/data/runme_post?_json=1", {}, csrftoken_from=True)

But without the csrftoken_from=True parameter it failed with the bindings error:

response = magic_parameters_client.post("/data/runme_post?_json=1", {})

Here's the test I wrote:

def test_magic_parameters_json_body(magic_parameters_client):
    magic_parameters_client.ds._metadata["databases"]["data"]["queries"]["runme_post"][
        "sql"
    ] = "insert into logs (line) values (:_header_host)"
    response = magic_parameters_client.post("/data/runme_post?_json=1", {}, csrftoken_from=True)
    assert response.status == 200
    assert response.json["ok"], response.json
    post_actual = magic_parameters_client.get(
        "/data/logs.json?_sort_desc=rowid&_shape=array"
    ).json[0]["line"]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries with magic parameters fail if POST body is empty 702069429  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 85.561ms · About: github-to-sqlite