html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/issues/2035#issuecomment-1460618433,https://api.github.com/repos/simonw/datasette/issues/2035,1460618433,IC_kwDOBm6k_c5XD0TB,9599,2023-03-08T18:06:34Z,2023-03-08T18:06:34Z,OWNER,"One way to do this would be to dynamically generate the `where id in (?, ?, ?)` with the correct number of question marks, then feed in a list from `request.args.getlist(""id"")` - but that would require rewriting the SQL query text to add those question marks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460621871,https://api.github.com/repos/simonw/datasette/issues/2035,1460621871,IC_kwDOBm6k_c5XD1Iv,9599,2023-03-08T18:08:25Z,2023-03-08T18:09:04Z,OWNER,"My current preferred solution is to lean into SQLite's JSON support. What if the query page spotted `?id=11&id=32&id=62` and turned that into a JSON string called `:id:` with a value of `[""11"", ""32"", ""62""]`? Note that this is still a string, not a list. This avoids a nasty problem that occurred in PHP world, where `?id[]=1&id[]=2` would result in an actual PHP array object, which often broke underlying code that had expected `$_GET[""id""]` to be a string, not an array. So in a query you'd be able to do this: where id in (select value from json_each(:id)) And then call it with `?id=11&id=32&id=62`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460628199,https://api.github.com/repos/simonw/datasette/issues/2035,1460628199,IC_kwDOBm6k_c5XD2rn,9599,2023-03-08T18:11:31Z,2023-03-08T18:11:31Z,OWNER,"One variant on this idea: maybe you have to specify in your query that you want it to be the JSON list version, not the single item (first `?id=` parameter version)? Maybe with syntax like this: where id in (select value from json_each(:id__list)) Datasette would automatically pass `{""id"": ""11"", ""id__list"": '[""11"", ""32"", ""62""]'}` as arguments to the `db.execute()` method, if the page was called with `?id=11&id=32&id=62`. This is more explicit, though the syntax is a bit uglier (maybe there's a nicer design for this?). I also worry about `?id__list=` conflicting with this, but I think that's a risk I can take - tell people not to do that, or even block `?id__list=` style parameters entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460632758,https://api.github.com/repos/simonw/datasette/issues/2035,1460632758,IC_kwDOBm6k_c5XD3y2,9599,2023-03-08T18:13:49Z,2023-03-08T18:13:49Z,OWNER,"https://github.com/rclement/datasette-dashboards/issues/54 makes the excellent point that the ` ``` Submitting that form with the middle two options selected navigates to: `https://www.example.com/?id=32&id=15`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460637906,https://api.github.com/repos/simonw/datasette/issues/2035,1460637906,IC_kwDOBm6k_c5XD5DS,9599,2023-03-08T18:16:31Z,2023-03-08T18:16:31Z,OWNER,"I'm pretty sold on this as a feature now. The main question I have is which of these options to implement: 1. `?id=1&?id=2` results in `:id` in the query being `[""1"", ""2""]` - no additional syntax required 2. `:id` in the query continues to reference just the first of those parameters - but `:id__list` (or some other custom syntax) instead gets `[""1"", ""2""]` - or, if the URL is `?id=1` - gets `[""1""]` Actually on writing these out I realize that option 2 is the ONLY valid option. It's no good building a query that works against a JSON list if the user might pass just a single ID, `?id=1`, resulting in their query breaking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460639749,https://api.github.com/repos/simonw/datasette/issues/2035,1460639749,IC_kwDOBm6k_c5XD5gF,9599,2023-03-08T18:17:31Z,2023-03-08T18:17:31Z,OWNER,"Since we are pre-1.0 it's still OK to implement a feature that disallows `?id__list=` in the URL, but allows `:id__list` in SQL queries to reference the JSON list of parameters. So I'm going to prototype this as the `:id__list` feature and see how it feels.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460654136,https://api.github.com/repos/simonw/datasette/issues/2035,1460654136,IC_kwDOBm6k_c5XD9A4,9599,2023-03-08T18:25:46Z,2023-03-08T18:25:46Z,OWNER,"Trickiest part of the implementation here is that it needs to know to output three `id` HTML form fields on the page, such that their values are persisted when the form is submitted a second time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460659382,https://api.github.com/repos/simonw/datasette/issues/2035,1460659382,IC_kwDOBm6k_c5XD-S2,9599,2023-03-08T18:28:00Z,2023-03-08T18:28:00Z,OWNER,"Also: `datasette-explain` may need to be updated to understand how to handle this: `ERROR: conn=, sql = 'explain select * from releases where id in (select id from json_each(:id__list))', params = None: You did not supply a value for binding parameter :id__list.` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460664619,https://api.github.com/repos/simonw/datasette/issues/2035,1460664619,IC_kwDOBm6k_c5XD_kr,9599,2023-03-08T18:32:29Z,2023-03-08T18:32:29Z,OWNER,"Got a prototype working: ```diff diff --git a/datasette/views/database.py b/datasette/views/database.py index 8d289105..6f9d8a44 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -226,6 +226,12 @@ class QueryView(DataView): ): db = await self.ds.resolve_database(request) database = db.name + # Disallow x__list query string parameters + invalid_params = [k for k in request.args if k.endswith(""__list"")] + if invalid_params: + raise DatasetteError( + ""Invalid query string parameters: {}"".format("", "".join(invalid_params)) + ) params = {key: request.args.get(key) for key in request.args} if ""sql"" in params: params.pop(""sql"") @@ -258,6 +264,11 @@ class QueryView(DataView): for named_parameter in named_parameters if not named_parameter.startswith(""_"") } + # Handle any __list parameters + for named_parameter in named_parameters: + if named_parameter.endswith(""__list""): + list_values = request.args.getlist(named_parameter[:-6]) + params[named_parameter] = json.dumps(list_values) # Set to blank string if missing from params for named_parameter in named_parameters: ``` This isn't yet doing the right thing on form re-submission: it breaks because it attempts to pass through the `?id__list=` invalid parameter. But I did manage to get it to do this through careful editing of the URL: That was this URL: `http://127.0.0.1:8034/content?sql=select+%3Aid__list%2C*+from+releases+where+id+in+(select+value+from+json_each(%3Aid__list))&id=62642726&id=18402901&id=38714866`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460668431,https://api.github.com/repos/simonw/datasette/issues/2035,1460668431,IC_kwDOBm6k_c5XEAgP,9599,2023-03-08T18:35:34Z,2023-03-08T18:35:34Z,OWNER,"To implement this properly need to do the following: - Get the page to display multiple `id: [ text input here ]` fields such that re-submission works - Figure out how this should work for canned queries and for writable canned queries - Tests that cover queries, canned queries, writable canned queries And a bonus feature: what if the Datasette UI layer spotted `:id__list` parameters and used them to add a bit of JavaScript that allowed users to click a `+` button next to an `id` form field to add another one? Also, when a page is re-displayed for on of these queries it could potentially add an extra form field allowing people to add another value. Though this has an annoying problem: how to tell the difference between an additional `id` input field that the user chose not to populate, v.s. one that is supposed to represent an empty string? Maybe only support multiple `id` fields for users with JavaScript in order to avoid this problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460679434,https://api.github.com/repos/simonw/datasette/issues/2035,1460679434,IC_kwDOBm6k_c5XEDMK,9599,2023-03-08T18:39:35Z,2023-03-08T18:39:35Z,OWNER,"I should consider the existing design of magic parameters here: https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters - `_actor_*` - `_header_*` - `_cookie_` - `_now_epoch` - `_now_date_utc` - `_now_datetime_utc` - `_random_chars_*` Should this new `id__list` syntax look more like those magic parameters, or is it OK to use `name__magic` syntax here instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460682625,https://api.github.com/repos/simonw/datasette/issues/2035,1460682625,IC_kwDOBm6k_c5XED-B,9599,2023-03-08T18:40:57Z,2023-03-08T18:40:57Z,OWNER,Pushed that prototype to a branch: https://github.com/simonw/datasette/commit/0fe844e9adb006a0138e83102ced1329d9155c59 / https://github.com/simonw/datasette/compare/sql-list-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1460808028,https://api.github.com/repos/simonw/datasette/issues/2035,1460808028,IC_kwDOBm6k_c5XEilc,1176293,2023-03-08T20:14:47Z,2023-03-08T20:14:47Z,NONE,"+1, I have been wishing for this feature (also for use with template-sql). It was requested before here #1304.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818, https://github.com/simonw/datasette/issues/2035#issuecomment-1492206593,https://api.github.com/repos/simonw/datasette/issues/2035,1492206593,IC_kwDOBm6k_c5Y8UQB,9599,2023-03-31T16:09:08Z,2023-03-31T16:09:08Z,OWNER,"I could ship this as part of: - #2049 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,