Codecov Report

Merging #1291 (8789157) into main (0a7621f) will not change coverage. The diff coverage is n/a.

:exclamation: Current head 8789157 differs from pull request most recent head 9bf089f. Consider uploading reports for the commit 9bf089f to get more accurate results

```diff @@ Coverage Diff @@

main #1291 +/-

======================================= Coverage 91.51% 91.51%
======================================= Files 34 34
Lines 4255 4255
======================================= Hits 3894 3894
Misses 361 361
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 0a7621f...9bf089f. Read the comment docs.

Relevant code is some of the most complex in all of Datasette.

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L530-L594

And

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L743-L771

I'll need to think hard about how to refactor this out into something more understandable before implementing previous links.

Same query again with a, d, v returns 0 results, which is also as we would want: it signifies that we are back to the very first page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=d&p2=v

I tried flipping the direction of the sort and the comparison operators and got this: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=h&p2=r

sql select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 < :p0) or (pk1 = :p0 and pk2 < :p1) or (pk1 = :p0 and pk2 = :p1 and pk3 < :p2)) order by pk1 desc, pk2 desc, pk3 desc limit 1 offset 99 Which returned a-d-v as desired. I messed around with it to find the limit 1 offset 99 values.

Let's figure out the SQL for this. The most complex case is probably this one: https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Ch%2Cr

Here's the SQL for that page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3E+%3Ap2%29%29+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=a&p1=h&p2=r

sql select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1) or (pk1 = :p0 and pk2 = :p1 and pk3 > :p2)) order by pk1, pk2, pk3 limit 101 Where p0 is a, p1 is h and p2 is r.

Given the above, how would I figure out the correct previous link? It should be https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv - a, d, v.

I think my ideal implementation for this would be to reverse the order, grab the previous page-size-plus-one items, then return a ?_next=x token that would provide the previous page sorted back in the expected default order.

The alternative would be to have a ?_previous=x token which can be used to paginate backwards in reverse order, but I think this would be confusing as it would result in "hit next page, then hit previous page" returning you to a new state which features rows in the reverse order.

https://twitter.com/llanga/status/1378431719934681094 looks like I should wait for 3.9.4, out in a few days.

For what it's worth, if anyone finds this in the future, I was having the same issue.

After digging through the code, it turned out that the database download is only available if it the db served in immutable mode, so datasette serve -i xyz.db rather than the doc's quickstart recommendation of datasette serve xyz.db.

I found one example of an implementation of reversed keyset pagination here: https://github.com/tvainika/objection-keyset-pagination/blob/cb21a493c96daa6e63c302efae6718d09aa11661/index.js#L74-L79

Codecov Report

Merging #1290 (2fb1e42) into main (87b583a) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #1290 +/-

======================================= Coverage 91.51% 91.51%
======================================= Files 34 34
Lines 4255 4255
======================================= Hits 3894 3894
Misses 361 361
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 87b583a...2fb1e42. Read the comment docs.

https://github.com/simonw/datasette/actions/runs/713207828 ran with pytest-xdist in 4m22s:

Here's the test suite running on regular pytest in 5m13s:

Not a huge speed-up because there are only 2 available cores in the GitHub Actions environment, but still worthwhile - especially since this lets people run in parallel on their own laptops.

Looks like -n auto only detected two cores on GitHub Actions: https://github.com/simonw/datasette/runs/2257597137?check_suite_focus=true ============================= test session starts ============================== platform linux -- Python 3.7.10, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 SQLite: 3.31.1 rootdir: /home/runner/work/datasette/datasette, configfile: pytest.ini plugins: xdist-2.2.1, timeout-1.4.2, forked-1.3.0, asyncio-0.14.0 gw0 I / gw1 I gw0 [878] / gw1 [878]

On my Mac pytest-xdist ran the test suite (minus two tests) in 59s, as opposed to 2m23s without xdist.