issue_comments

Relevant code is some of the most complex in all of Datasette.

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L530-L594

And

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L743-L771

I'll need to think hard about how to refactor this out into something more understandable before implementing previous links.

Same query again with a, d, v returns 0 results, which is also as we would want: it signifies that we are back to the very first page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=d&p2=v

I tried flipping the direction of the sort and the comparison operators and got this: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=h&p2=r

sql select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 < :p0) or (pk1 = :p0 and pk2 < :p1) or (pk1 = :p0 and pk2 = :p1 and pk3 < :p2)) order by pk1 desc, pk2 desc, pk3 desc limit 1 offset 99 Which returned a-d-v as desired. I messed around with it to find the limit 1 offset 99 values.

Let's figure out the SQL for this. The most complex case is probably this one: https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Ch%2Cr

Here's the SQL for that page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3E+%3Ap2%29%29+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=a&p1=h&p2=r

sql select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1) or (pk1 = :p0 and pk2 = :p1 and pk3 > :p2)) order by pk1, pk2, pk3 limit 101 Where p0 is a, p1 is h and p2 is r.

Given the above, how would I figure out the correct previous link? It should be https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv - a, d, v.

I think my ideal implementation for this would be to reverse the order, grab the previous page-size-plus-one items, then return a ?_next=x token that would provide the previous page sorted back in the expected default order.

The alternative would be to have a ?_previous=x token which can be used to paginate backwards in reverse order, but I think this would be confusing as it would result in "hit next page, then hit previous page" returning you to a new state which features rows in the reverse order.

https://twitter.com/llanga/status/1378431719934681094 looks like I should wait for 3.9.4, out in a few days.

I found one example of an implementation of reversed keyset pagination here: https://github.com/tvainika/objection-keyset-pagination/blob/cb21a493c96daa6e63c302efae6718d09aa11661/index.js#L74-L79

https://github.com/simonw/datasette/actions/runs/713207828 ran with pytest-xdist in 4m22s:

Here's the test suite running on regular pytest in 5m13s:

Not a huge speed-up because there are only 2 available cores in the GitHub Actions environment, but still worthwhile - especially since this lets people run in parallel on their own laptops.

Looks like -n auto only detected two cores on GitHub Actions: https://github.com/simonw/datasette/runs/2257597137?check_suite_focus=true ============================= test session starts ============================== platform linux -- Python 3.7.10, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 SQLite: 3.31.1 rootdir: /home/runner/work/datasette/datasette, configfile: pytest.ini plugins: xdist-2.2.1, timeout-1.4.2, forked-1.3.0, asyncio-0.14.0 gw0 I / gw1 I gw0 [878] / gw1 [878]

On my Mac pytest-xdist ran the test suite (minus two tests) in 59s, as opposed to 2m23s without xdist.