{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-650684635", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 650684635, "node_id": "MDEyOklzc3VlQ29tbWVudDY1MDY4NDYzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-28T03:30:31Z", "updated_at": "2020-06-28T03:30:31Z", "author_association": "OWNER", "body": "Live demo: https://latest.datasette.io/fixtures/magic_parameters", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-650679100", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 650679100, "node_id": "MDEyOklzc3VlQ29tbWVudDY1MDY3OTEwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-28T03:00:44Z", "updated_at": "2020-06-28T03:00:44Z", "author_association": "OWNER", "body": "I'm going to add some canned queries to the `metadata.json` used by the live demo that illustrate this feature.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-650678951", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 650678951, "node_id": "MDEyOklzc3VlQ29tbWVudDY1MDY3ODk1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-28T02:59:52Z", "updated_at": "2020-06-28T02:59:52Z", "author_association": "OWNER", "body": "Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#magic-parameters\r\n\r\nPlugin hook documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#plugin-hook-register-magic-parameters", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-650648434", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 650648434, "node_id": "MDEyOklzc3VlQ29tbWVudDY1MDY0ODQzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-27T23:27:35Z", "updated_at": "2020-06-27T23:37:38Z", "author_association": "OWNER", "body": "I'm going to rename `_request_X` to `_header_X` as that better reflects what it now does.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-650593122", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 650593122, "node_id": "MDEyOklzc3VlQ29tbWVudDY1MDU5MzEyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-27T18:03:02Z", "updated_at": "2020-06-27T18:03:10Z", "author_association": "OWNER", "body": "> Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.\r\n\r\nThis is an open security hole in https://github.com/simonw/datasette/commit/94c1315f0030fd58ce46a9294052c5c9d9d181c7 - it's useful for testing, but I need to remove it before I land that branch.\r\n\r\nhttps://github.com/simonw/datasette/blob/94c1315f0030fd58ce46a9294052c5c9d9d181c7/datasette/views/database.py#L231-L237\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-650458857", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 650458857, "node_id": "MDEyOklzc3VlQ29tbWVudDY1MDQ1ODg1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-27T00:11:04Z", "updated_at": "2020-06-27T00:11:04Z", "author_association": "OWNER", "body": "Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-650455793", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 650455793, "node_id": "MDEyOklzc3VlQ29tbWVudDY1MDQ1NTc5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-26T23:57:30Z", "updated_at": "2020-06-27T00:00:16Z", "author_association": "OWNER", "body": "Maybe I should ship a default `_scope_headers_...` parameter instead, which reads from a dictionary of `scope[\"headers\"]` - https://asgi-scope.now.sh/ shows what those look like.\r\n\r\n```\r\n{'client': ('148.64.98.14', 0),\r\n 'headers': [[b'host', b'asgi-scope.now.sh'],\r\n             [b'x-forwarded-for', b'148.64.98.14'],\r\n             [b'x-vercel-id', b'sw72x-1593215573008-024e4e603806'],\r\n             [b'x-forwarded-host', b'asgi-scope.now.sh'],\r\n             [b'accept',\r\n              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'\r\n              b'webp,*/*;q=0.8'],\r\n             [b'x-real-ip', b'148.64.98.14'],\r\n             [b'x-vercel-deployment-url', b'asgi-scope-9eyeojbek.now.sh'],\r\n             [b'upgrade-insecure-requests', b'1'],\r\n             [b'x-vercel-trace', b'sfo1'],\r\n             [b'x-forwarded-proto', b'https'],\r\n             [b'accept-language', b'en-US,en;q=0.5'],\r\n             [b'user-agent',\r\n              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko'\r\n              b'/20100101 Firefox/77.0'],\r\n             [b'x-vercel-forwarded-for', b'148.64.98.14'],\r\n             [b'accept-encoding', b'gzip, deflate, br'],\r\n             [b'dnt', b'1'],\r\n             [b'te', b'trailers']],\r\n 'http_version': '1.1',\r\n 'method': 'GET',\r\n 'path': '/',\r\n 'query_string': b'',\r\n 'raw_path': b'/',\r\n 'root_path': '',\r\n 'scheme': 'https',\r\n 'server': ('asgi-scope.now.sh', 80),\r\n 'type': 'http'}\r\n```\r\n\r\nI'm going to have `_request_X` actually mean \"find the first value for X in `scope[\"headers\"`]\" - with underscores converted to hyphens.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-650455353", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 650455353, "node_id": "MDEyOklzc3VlQ29tbWVudDY1MDQ1NTM1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-26T23:55:40Z", "updated_at": "2020-06-26T23:55:40Z", "author_association": "OWNER", "body": "`_request_ip` is actually quite hard to implement - should it take into account things like the `x-forwarded-for` header?\r\n\r\nIt probably should - but that means it now needs a bunch of extra configuration to tell it which of those headers can be trusted in the current environment.\r\n\r\nAs such I think I'll leave that for a plugin.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-649014757", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 649014757, "node_id": "MDEyOklzc3VlQ29tbWVudDY0OTAxNDc1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-24T19:15:46Z", "updated_at": "2020-06-24T19:31:52Z", "author_association": "OWNER", "body": "I'm building this documentation-first - here's the documentation so far: https://github.com/simonw/datasette/blob/6fc8bd9c473f4a25e0a076f24c7e5a9b2f353bb8/docs/sql_queries.rst#magic-parameters", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-646271834", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 646271834, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NjI3MTgzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-18T19:49:41Z", "updated_at": "2020-06-24T18:49:22Z", "author_association": "OWNER", "body": "But then what kind of magic parameters might plugins want to add?\r\n\r\nHere's a crazy idea: `_scrapedcontent_url` - it would look for the `url` column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function `scrapedcontent()` would need to optionally be sent the full row dictionary being inserted too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-646270702", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 646270702, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NjI3MDcwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-18T19:47:19Z", "updated_at": "2020-06-24T18:48:48Z", "author_association": "OWNER", "body": "Brainstorming more potential magic parameters:\r\n\r\n* `_actor_id`\r\n* `_actor_name`\r\n* `_request_ip`\r\n* `_request_user_agent`\r\n* `_cookie_cookiename`\r\n* `_signedcookie_cookiename` - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, `_signedcookie_cookiename_mynamespace`. Might not need special signed cookie support since `actor` is already usually from a signed cookie.\r\n* `_timestamp_unix` (not happy with these names yet)\r\n* `_timestamp_localtime`\r\n* `_timestamp_datetime`\r\n* `_timestamp_utc`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-649000075", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 649000075, "node_id": "MDEyOklzc3VlQ29tbWVudDY0OTAwMDA3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-24T18:46:36Z", "updated_at": "2020-06-24T18:47:37Z", "author_association": "OWNER", "body": "Another magic parameter that would be useful would be `_random`. Consider https://github.com/simonw/datasette-auth-tokens/issues/1 for example - I'd like to be able to provide a writable canned query which can create new authentication tokens in the database, but ideally it would automatically populate a secure random secret for each one.\r\n\r\nMaybe `_random_chars_128` to create a 128 character long random string (using `os.urandom(64).hex()`).\r\n\r\nThis would be the first example of a magic parameter where part of the parameter name is used to configure the resulting value. Maybe neater to separate that with a different character? Unfortunately `_random_chars:128` wouldn't work because these parameters are used in a SQLite query where `:` has special meaning: `insert into blah (secret) values (:_random_chars:128)` wouldn't make sense.\r\n\r\nActually this is already supported by the proposed design - `_random_chars_128` would become `random(\"chars_128\")` so the `random()` function could split off the 128 itself.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-646272627", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 646272627, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NjI3MjYyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-18T19:51:32Z", "updated_at": "2020-06-18T19:51:32Z", "author_association": "OWNER", "body": "I'd be OK with the first version of this not including a plugin hook.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-646264051", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 646264051, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NjI2NDA1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-18T19:32:13Z", "updated_at": "2020-06-18T19:32:37Z", "author_association": "OWNER", "body": "If every magic parameter has a prefix and suffix, like `_request_ip` and `_actor_id`, then plugins could register a function for a prefix. Register a function to `_actor` and `actor(\"id\")`will be called for `_actor_id`.\r\n\r\nBut does it make sense for every magic parameter to be of form `_a_b`? I think so.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-646246062", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 646246062, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NjI0NjA2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-18T18:54:41Z", "updated_at": "2020-06-18T18:54:41Z", "author_association": "OWNER", "body": "The `_actor_id` param makes this a bit trickier, because we can't just say \"if you see an unknown parameter called X call this function\" - our magic parameter logic isn't adding single parameters, it might add a whole family of them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-646242172", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 646242172, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NjI0MjE3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-18T18:46:06Z", "updated_at": "2020-06-18T18:53:31Z", "author_association": "OWNER", "body": "Yes that can work - and using `__missing__` (new in Python 3) is nicer because then the regular dictionary gets checked first:\r\n```python\r\nimport sqlite3\r\n\r\nconn = sqlite3.connect(\":memory:\")\r\n\r\n\r\nclass Magic(dict):\r\n    def __missing__(self, key):\r\n        return key.upper()\r\n\r\n\r\nconn.execute(\"select :name\", Magic()).fetchall()\r\n```\r\nOutputs:\r\n```\r\n[('NAME',)]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-646238702", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 646238702, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NjIzODcwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-18T18:39:07Z", "updated_at": "2020-06-18T18:39:07Z", "author_association": "OWNER", "body": "It would be nice if Datasette didn't have to do any additional work to find e.g. `_request_ip` if that parameter turned out not to be used by the query.\r\n\r\nCould I do this with a custom class that implements `__getitem__()` and then gets passed as SQLite arguments?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/842#issuecomment-643663005", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/842", "id": 643663005, "node_id": "MDEyOklzc3VlQ29tbWVudDY0MzY2MzAwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-13T18:51:57Z", "updated_at": "2020-06-13T18:51:57Z", "author_association": "OWNER", "body": "Two potential designs:\r\n\r\n- `_actor_id`, `_request_ip`, `_now_timestamp` - so special reserved parameters\r\n- a SQL function: `update blah set up = special('ip')`\r\n\r\nI fee the first would be easier to implement.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638212085, "label": "Magic parameters for canned queries"}, "performed_via_github_app": null}