issue_comments

I could ship this as part of: - #2049

Pushed that prototype to a branch: https://github.com/simonw/datasette/commit/0fe844e9adb006a0138e83102ced1329d9155c59 / https://github.com/simonw/datasette/compare/sql-list-parameters

I should consider the existing design of magic parameters here: https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters

• _actor_*
• _header_*
• _cookie_
• _now_epoch
• _now_date_utc
• _now_datetime_utc
• _random_chars_*

Should this new id__list syntax look more like those magic parameters, or is it OK to use name__magic syntax here instead?

To implement this properly need to do the following: - Get the page to display multiple id: [ text input here ] fields such that re-submission works - Figure out how this should work for canned queries and for writable canned queries - Tests that cover queries, canned queries, writable canned queries

And a bonus feature: what if the Datasette UI layer spotted :id__list parameters and used them to add a bit of JavaScript that allowed users to click a + button next to an id form field to add another one?

Also, when a page is re-displayed for on of these queries it could potentially add an extra form field allowing people to add another value.

Though this has an annoying problem: how to tell the difference between an additional id input field that the user chose not to populate, v.s. one that is supposed to represent an empty string?

Maybe only support multiple id fields for users with JavaScript in order to avoid this problem.

Got a prototype working: ```diff diff --git a/datasette/views/database.py b/datasette/views/database.py index 8d289105..6f9d8a44 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -226,6 +226,12 @@ class QueryView(DataView): ): db = await self.ds.resolve_database(request) database = db.name + # Disallow x__list query string parameters + invalid_params = [k for k in request.args if k.endswith("__list")] + if invalid_params: + raise DatasetteError( + "Invalid query string parameters: {}".format(", ".join(invalid_params)) + ) params = {key: request.args.get(key) for key in request.args} if "sql" in params: params.pop("sql") @@ -258,6 +264,11 @@ class QueryView(DataView): for named_parameter in named_parameters if not named_parameter.startswith("_") } + # Handle any __list parameters + for named_parameter in named_parameters: + if named_parameter.endswith("__list"): + list_values = request.args.getlist(named_parameter[:-6]) + params[named_parameter] = json.dumps(list_values)

     # Set to blank string if missing from params
     for named_parameter in named_parameters:

`` This isn't yet doing the right thing on form re-submission: it breaks because it attempts to pass through the?id__list=` invalid parameter. But I did manage to get it to do this through careful editing of the URL:

That was this URL: http://127.0.0.1:8034/content?sql=select+%3Aid__list%2C*+from+releases+where+id+in+(select+value+from+json_each(%3Aid__list))&id=62642726&id=18402901&id=38714866

Also: datasette-explain may need to be updated to understand how to handle this:

ERROR: conn=<sqlite3.Connection object at 0x102834940>, sql = 'explain select * from releases where id in (select id from json_each(:id__list))', params = None: You did not supply a value for binding parameter :id__list.

Trickiest part of the implementation here is that it needs to know to output three id HTML form fields on the page, such that their values are persisted when the form is submitted a second time.

Since we are pre-1.0 it's still OK to implement a feature that disallows ?id__list= in the URL, but allows :id__list in SQL queries to reference the JSON list of parameters.

So I'm going to prototype this as the :id__list feature and see how it feels.

I'm pretty sold on this as a feature now. The main question I have is which of these options to implement:

1. ?id=1&?id=2 results in :id in the query being ["1", "2"] - no additional syntax required
2. :id in the query continues to reference just the first of those parameters - but :id__list (or some other custom syntax) instead gets ["1", "2"] - or, if the URL is ?id=1 - gets ["1"]

Actually on writing these out I realize that option 2 is the ONLY valid option. It's no good building a query that works against a JSON list if the user might pass just a single ID, ?id=1, resulting in their query breaking.

https://github.com/rclement/datasette-dashboards/issues/54 makes the excellent point that the <select multiple> default HTML widget produces this exact format of query string:

```html

`` Submitting that form with the middle two options selected navigates to:https://www.example.com/?id=32&id=15`

One variant on this idea: maybe you have to specify in your query that you want it to be the JSON list version, not the single item (first ?id= parameter version)? Maybe with syntax like this:

where id in (select value from json_each(:id__list))

Datasette would automatically pass {"id": "11", "id__list": '["11", "32", "62"]'} as arguments to the db.execute() method, if the page was called with ?id=11&id=32&id=62.

This is more explicit, though the syntax is a bit uglier (maybe there's a nicer design for this?). I also worry about ?id__list= conflicting with this, but I think that's a risk I can take - tell people not to do that, or even block ?id__list= style parameters entirely.

My current preferred solution is to lean into SQLite's JSON support.

What if the query page spotted ?id=11&id=32&id=62 and turned that into a JSON string called :id: with a value of ["11", "32", "62"]?

Note that this is still a string, not a list. This avoids a nasty problem that occurred in PHP world, where ?id[]=1&id[]=2 would result in an actual PHP array object, which often broke underlying code that had expected $_GET["id"] to be a string, not an array.

So in a query you'd be able to do this:

where id in (select value from json_each(:id))

And then call it with ?id=11&id=32&id=62.

One way to do this would be to dynamically generate the where id in (?, ?, ?) with the correct number of question marks, then feed in a list from request.args.getlist("id") - but that would require rewriting the SQL query text to add those question marks.