issue_comments

It's interesting to note WHY the time limit works against this so well.

The time limit as-implemented looks like this:

https://github.com/simonw/datasette/blob/5f9f567acbc58c9fcd88af440e68034510fb5d2b/datasette/utils/init.py#L181-L201

The key here is conn.set_progress_handler(handler, n) - which specifies that the handler function should be called every n SQLite operations.

The handler function then checks to see if too much time has transpired and conditionally cancels the query.

This also doubles up as a "maximum number of operations" guard, which is what's happening when you attempt to fetch an infinite number of rows from an infinite table.

That limit code could even be extended to say "exit the query after either 5s or 50,000,000 operations".

I don't think that's necessary though.

To be honest I'm having trouble with the idea of dropping max_returned_rows mainly because what Datasette does (allow arbitrary untrusted SQL queries) is dangerous, so I've designed in multiple redundant defence-in-depth mechanisms right from the start.

It would be really neat if we could explore this idea in a plugin, but I don't think Datasette has plugin hooks in the right place for that at the moment.

Yes good point, the time limit does already protect against that. I've been contemplating a permissioned-users-only relaxation of that time limit too, and I got that idea mixed up with this one in my head.

On that basis maybe this feature would be safe after all? Would need to do some testing, but it may be that the existing time limit provides enough protection here already.

That recursive query is a great example of the kind of thing having a maximum row limit protects against.

Imagine if Datasette CSVs did allow unlimited retrievals. Someone could hit the CSV endpoint for that recursive query and tie up Datasette's SQL connection effectively forever.

Even if this feature becomes a permission-guarded thing we still need to take that case into account.

At the very least it would be good if the query could be cancelled if the client disconnects - so if someone accidentally starts an infinite query they can cancel the request and free up the server resources.

It might be a good idea to implement a page that shows "currently running" queries and allows users with the right permission to terminate them from that page.

Another option: a "limit of last resource" - either a very high row limit (10,000,000 perhaps) or even a time limit, saying that all queries will be cancelled if they take longer than thirty minutes or similar.

The protection is supposed to be from this line: python rows = cursor.fetchmany(max_returned_rows + 1) By capping the call to .fetchman() at max_returned_rows + 1 (the + 1 is to allow detection of whether or not there is a next page) I'm ensuring that Datasette never attempts to iterate over a huge result set.

SQLite and the sqlite3 library seem to handle this correctly. Here's an example:

```pycon

import sqlite3 conn = sqlite3.connect(":memory:") cursor = conn.execute(""" ... with recursive counter(x) as ( ... select 0 ... union ... select x + 1 from counter ... ) ... select * from counter""") cursor.fetchmany(10) [(0,), (1,), (2,), (3,), (4,), (5,), (6,), (7,), (8,), (9,), (10,)] ``counter` there is an infinitely long table (see TIL) - but we can retrieve the first 10 results without going into an infinite loop.

My main concern here is that public Datasette instances could easily have all of their available database connections consumed by long-running queries - either accidentally or deliberately.

I do totally understand the need for this feature though. I think it can absolutely make sense provided it's protected by authentication and permissions.

Maybe even limit the number of concurrent downloads at once such that there's always at least one database connection free for other requests.