{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692951144", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692951144, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk1MTE0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:08:12Z", "updated_at": "2020-09-15T20:08:12Z", "author_association": "OWNER", "body": "I think the easiest fix is for me to ensure that calls to `__len__` on the `MagicParameters` class always return at least 1.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692946616", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692946616, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk0NjYxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:59:21Z", "updated_at": "2020-09-15T19:59:21Z", "author_association": "OWNER", "body": "I wish I could call https://www.sqlite.org/c3ref/bind_parameter_count.html and https://www.sqlite.org/c3ref/bind_parameter_name.html from Python.\r\n\r\nMight be possible to do that using `ctypes` - see this example code: https://mail.python.org/pipermail//pypy-commit/2013-February/071372.html\r\n\r\n```python\r\n            param_count = lib.sqlite3_bind_parameter_count(self.statement)\r\n            for idx in range(1, param_count + 1):\r\n                param_name = lib.sqlite3_bind_parameter_name(self.statement,\r\n                                                             idx)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692945504", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692945504, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk0NTUwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:57:10Z", "updated_at": "2020-09-15T19:57:10Z", "author_association": "OWNER", "body": "So the problem actually occurs when the `MagicParameters` class wraps an empty dictionary.\r\n\r\nRelevant code:\r\n\r\nhttps://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L228-L236\r\n\r\nAnd:\r\n\r\nhttps://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L364-L383\r\n\r\nI'm passing a special magic parameters dictionary for the Python `sqlite3` module to look up parameters in. When that dictionary is `{}` a `__len__` check is performed on that dictionary, the result comes back as 0 and as a result it assumes there are no parameters.\r\n\r\nI tracked down the relevant C code:\r\n\r\nhttps://github.com/python/cpython/blob/81715808716198471fbca0a3db42ac408468dbc5/Modules/_sqlite/statement.c#L218-L237\r\n\r\n```c\r\n    Py_BEGIN_ALLOW_THREADS\r\n    num_params_needed = sqlite3_bind_parameter_count(self->st);\r\n    Py_END_ALLOW_THREADS\r\n\r\n    if (PyTuple_CheckExact(parameters) || PyList_CheckExact(parameters) || (!PyDict_Check(parameters) && PySequence_Check(parameters))) {\r\n        /* parameters passed as sequence */\r\n        if (PyTuple_CheckExact(parameters)) {\r\n            num_params = PyTuple_GET_SIZE(parameters);\r\n        } else if (PyList_CheckExact(parameters)) {\r\n            num_params = PyList_GET_SIZE(parameters);\r\n        } else {\r\n            num_params = PySequence_Size(parameters);\r\n        }\r\n        if (num_params != num_params_needed) {\r\n            PyErr_Format(pysqlite_ProgrammingError,\r\n                         \"Incorrect number of bindings supplied. The current \"\r\n                         \"statement uses %d, and there are %zd supplied.\",\r\n                         num_params_needed, num_params);\r\n            return;\r\n        }\r\n```\r\n\r\nIt looks to me like this should fail if the number of keys known to be in the dictionary differs from the number of named parameters in the query. But if those numbers fail to match it still works as far as I can tell - it's only dictionary length of 0 that is causing the problems.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692940375", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692940375, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk0MDM3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:47:09Z", "updated_at": "2020-09-15T19:47:09Z", "author_association": "OWNER", "body": "Yes! The tests all pass if I update the test function to do this:\r\n```python\r\n    response = magic_parameters_client.post(\r\n        \"/data/runme_post{}\".format(qs),\r\n        {\"ignore_me\": \"1\"},\r\n        csrftoken_from=use_csrf or None,\r\n        allow_redirects=False,\r\n    )\r\n```\r\nSo the bug only occurs if the POST body is completely empty.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692938935", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692938935, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjkzODkzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:44:21Z", "updated_at": "2020-09-15T19:44:41Z", "author_association": "OWNER", "body": "While I'm running the above test, in the rounds that work the `receive()` awaitable returns `{'type': 'http.request', 'body': b'csrftoken=IlpwUGlSMFVVa3Z3ZlVoamQi.uY2U1tF4i0M-5M6x34vnBCmJgr0'}`\r\n\r\nIn the rounds that fails it returns `{'type': 'http.request'}`\r\n\r\nSo it looks like the `csrftoken_from=True` parameter may be helping just by ensuring the `body` key is present and not missing. I wonder if it would work if a body of `b''` was present there?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692937150", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692937150, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjkzNzE1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:42:57Z", "updated_at": "2020-09-15T19:42:57Z", "author_association": "OWNER", "body": "New (failing) test:\r\n```python\r\n@pytest.mark.parametrize(\"use_csrf\", [True, False])\r\n@pytest.mark.parametrize(\"return_json\", [True, False])\r\ndef test_magic_parameters_csrf_json(magic_parameters_client, use_csrf, return_json):\r\n    magic_parameters_client.ds._metadata[\"databases\"][\"data\"][\"queries\"][\"runme_post\"][\r\n        \"sql\"\r\n    ] = \"insert into logs (line) values (:_header_host)\"\r\n    qs = \"\"\r\n    if return_json:\r\n        qs = \"?_json=1\"\r\n    response = magic_parameters_client.post(\r\n        \"/data/runme_post{}\".format(qs),\r\n        {},\r\n        csrftoken_from=use_csrf or None,\r\n        allow_redirects=False,\r\n    )\r\n    if return_json:\r\n        assert response.status == 200\r\n        assert response.json[\"ok\"], response.json\r\n    else:\r\n        assert response.status == 302\r\n        messages = magic_parameters_client.ds.unsign(\r\n            response.cookies[\"ds_messages\"], \"messages\"\r\n        )\r\n        assert [[\"Query executed, 1 row affected\", 1]] == messages\r\n    post_actual = magic_parameters_client.get(\r\n        \"/data/logs.json?_sort_desc=rowid&_shape=array\"\r\n    ).json[0][\"line\"]\r\n    assert post_actual == \"localhost\"\r\n```\r\nIt passes twice, fails twice - failures are for the ones where `use_csrf` is `False`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692927867", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692927867, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjkyNzg2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:25:23Z", "updated_at": "2020-09-15T19:25:23Z", "author_association": "OWNER", "body": "Hunch: I think the `asgi-csrf` middleware may be consuming the request body and failing to restore it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692835066", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692835066, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjgzNTA2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T16:40:12Z", "updated_at": "2020-09-15T16:40:12Z", "author_association": "OWNER", "body": "Is the bug here that magic parameters are incompatible with CSRF-exempt requests (e.g. request with no cookies)?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692834670", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692834670, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjgzNDY3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T16:39:29Z", "updated_at": "2020-09-15T16:39:29Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L222-L236\r\n\r\nThis issue may not be about `_json=1` interacting with magic parameters after all.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692834064", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692834064, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjgzNDA2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T16:38:21Z", "updated_at": "2020-09-15T16:38:21Z", "author_association": "OWNER", "body": "So the mystery here is why does omitting `csrftoken_from=True` break the `MagicParameters` mechanism?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692832113", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692832113, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjgzMjExMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T16:34:53Z", "updated_at": "2020-09-15T16:37:43Z", "author_association": "OWNER", "body": "This is so weird. In the test I wrote for this the following passed:\r\n\r\n    response = magic_parameters_client.post(\"/data/runme_post?_json=1\", {}, csrftoken_from=True)\r\n\r\nBut without the `csrftoken_from=True` parameter it failed with the bindings error:\r\n\r\n    response = magic_parameters_client.post(\"/data/runme_post?_json=1\", {})\r\n\r\nHere's the test I wrote:\r\n\r\n```python\r\ndef test_magic_parameters_json_body(magic_parameters_client):\r\n    magic_parameters_client.ds._metadata[\"databases\"][\"data\"][\"queries\"][\"runme_post\"][\r\n        \"sql\"\r\n    ] = \"insert into logs (line) values (:_header_host)\"\r\n    response = magic_parameters_client.post(\"/data/runme_post?_json=1\", {}, csrftoken_from=True)\r\n    assert response.status == 200\r\n    assert response.json[\"ok\"], response.json\r\n    post_actual = magic_parameters_client.get(\r\n        \"/data/logs.json?_sort_desc=rowid&_shape=array\"\r\n    ).json[0][\"line\"]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null}