issue_comments

Amusing edge-case: if you run this against a explain ... query it falls back to using regular expressions, because explain explain select ... is invalid SQL. https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=

SQLite carries a warning about using EXPLAIN like this: https://www.sqlite.org/lang_explain.html

The output from EXPLAIN and EXPLAIN QUERY PLAN is intended for interactive analysis and troubleshooting only. The details of the output format are subject to change from one release of SQLite to the next. Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.

I think that's OK here, because of the regular expression fallback. If the format changes in the future in a way that breaks the query the error should be caught and the regex-captured parameters should be returned instead.

Hmmm... actually that's not entirely true:

https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/init.py#L1084-L1091

If the format changes such that the same columns are returned but the [row["p4"].lstrip(":") for row in results if row["opcode"] == "Variable"] list comprehension returns an empty array it will break Datasette!

I'm going to take that risk for the moment, but I'll actively watch out for problems in the future. If this does turn out to be bad I can always go back to the pure regular expression mechanism.

Fixed! Fantastic, this one has been bothering me for years.

https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27

One catch with this approach: if the SQL query is invalid, the parameters will not be extracted and shown as form fields.

Maybe that's completely fine? Why display a form if it's going to break when the user actually runs the query?

But it does bother me. I worry that someone who is still iterating on and editing their query before actually starting to use it might find the behaviour confusing.

So maybe if the query raises an exception it could fall back on the regular expression results?

I think this works!

```python re_named_parameter = re.compile(":([a-zA-Z0-9]+)")

async def derive_named_parameters(db, sql): explain = 'explain {}'.format(sql.strip().rstrip(";")) possible_params = _re_named_parameter.findall(sql) try: results = await db.execute(explain, {p: None for p in possible_params}) return [row["p4"].lstrip(":") for row in results if row["opcode"] == "Variable"] except sqlite3.DatabaseError: return [] ```

This may not work:

ERROR: sql = 'explain select 1 + :one + :two', params = None: You did not supply a value for binding 1.

The explain queries themselves want me to pass them parameters.

I could try using the regex to pull out candidates and passing None for each of those, including incorrect ones like :31.

Relevant code: https://github.com/simonw/datasette/blob/ad90a72afa21b737b162e2bbdddc301a97d575cd/datasette/views/database.py#L225-L231

I hoped this would work: sql with foo as ( explain select * from facetable where state = :state and on_earth = :on_earth and neighborhood not like '00:04' ) select p4 from foo where opcode = 'Variable' But sadly it returns an error:

near "explain": syntax error

I may have a way to work around this, using explain. Consider this query:

sql select * from facetable where state = :state and on_earth = :on_earth and neighborhood not like '00:04' Datasette currently gets confused and shows three form fields: https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04=

But... if I run explain against that I get this (truncated):

addr | opcode | p1 | p2 | p3 | p4 | p5 | comment -- | -- | -- | -- | -- | -- | -- | -- 20 | ResultRow | 6 | 10 | 0 |   | 0 |   21 | Next | 0 | 3 | 0 |   | 1 |   22 | Halt | 0 | 0 | 0 |   | 0 |   23 | Transaction | 0 | 0 | 35 | 0 | 1 |   24 | Variable | 1 | 2 | 0 | :state | 0 |   25 | Variable | 2 | 3 | 0 | :on_earth | 0 |   26 | String8 | 0 | 4 | 0 | 00:04 | 0 |   27 | Goto | 0 | 1 | 0 |   | 0 |  

Could it be as simple as pulling out those Variable rows to figure out the names of the variables in the query?

Marking this blocked because I don't have a way around the needing-a-SQLite-SQL-parser problem at the moment.

Urgh, yeah I've seen this one before. Fixing it pretty much requires writing a full SQLite SQL syntax parser in Python, which is frustratingly complicated for solving this issue!

You can work around this for a canned query by using the optional params: argument documented here: https://docs.datasette.io/en/stable/sql_queries.html#canned-query-parameters