issue_comments

Actually next page SQL when sorting looks more like this:

select rowid, * from [alcohol-consumption/drinks] where "country" like :p0 and ( beer_servings > 111 or (beer_servings = 111 and rowid > 190) ) order by beer_servings, rowid limit 101

The next page after row 190 with sortable value 111 should show either records that are greater than 111 or records that match 111 but have a greater primary key than the last one seen.

https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=select+rowid%2C+*+from+%5Balcohol-consumption%2Fdrinks%5D%0D%0Awhere+%22country%22+like+%3Ap0%0D%0Aand+%28%0D%0A++++beer_servings+%3E+111%0D%0A++++or+%28beer_servings+%3D+111+and+rowid+%3E+190%29%0D%0A%29%0D%0Aorder+by+beer_servings%2C+rowid+limit+101&p0=%25a%25

To break this up into smaller units, the first implementation of this will only support a single _sort or _sort_desc querystring parameter.

Could also identify all views for that database, which would save on these queries:

https://github.com/simonw/datasette/blob/b2188f044265c95f7e54860e28107c17d2a6ed2e/datasette/app.py#L543-L545

From a code point of view, the current mechanism for _group_count makes the TableView even more complicated:

https://github.com/simonw/datasette/blob/446d47fdb005b3776bc06ad8d1f44b01fc2e938b/datasette/app.py#L644-L653

Instead, I think if _group_count is detected we should generate the SQL and then defer to self.custom_sql, like we do for canned queries:

https://github.com/simonw/datasette/blob/446d47fdb005b3776bc06ad8d1f44b01fc2e938b/datasette/app.py#L539-L541

The single biggest challenge here is expanding foreign key references. This is the blocker that prevents _group_count from being useful at the moment.

While I'm at it, doing the same thing for fts_table detection is worth considering:

https://github.com/simonw/datasette/blob/446d47fdb005b3776bc06ad8d1f44b01fc2e938b/datasette/app.py#L598-L603

A note about views: a view cannot be paginated using keyset pagination because records returned from a view don't have a primary key - so there's no way to reliably distinguish between _next= records when the sorted column has duplicates with the same value.

Datasette already takes this into account: views are paginated using offset/limit instead. We can continue to do that even for views that have been sorted using a _sort parameter.

https://github.com/simonw/datasette/blob/446d47fdb005b3776bc06ad8d1f44b01fc2e938b/datasette/app.py#L93-L102

table_rows_count is always the total number of rows in the table.

Maybe table_rows_filtered_count would be more aesthetically pleasing than filtered_table_rows_count.

A common problem with keyset pagination is that it can distort the "total number of rows" logic - every time you navigate to a further page the total rows count can decrease due to the extra arguments in the where clause. The filtered_table_rows value (see #194) calculated using count_sql currently has this problem.

It would be useful to have a microbenchmark in place to help understand how much of a performance benefit this would actually provide.

I'm going to combine the code for explicit sorting with the existing code for _next= pagination - so even tables without an explicit sort order will run through the same code since they are ordered and paginated by primary key.