issue_comments

Here's the SQL for a next page with three compound primary keys:

https://datasette-issue-190-compound-pks.now.sh/compound-pks-8e99805?sql=select+*+from+compound_three_primary_keys%0D%0Awhere%0D%0A++%28pk1+%3E+%3Apk1%29%0D%0A++++or%0D%0A++%28pk1+%3D+%3Apk1+and+pk2+%3E+%3Apk2%29%0D%0A++++or%0D%0A++%28pk1+%3D+%3Apk1+and+pk2+%3D+%3Apk2+and+pk3+%3E+%3Apk3%29%0D%0Aorder+by+pk1%2C+pk2%2C+pk3%3B%0D%0A%0D%0A%0D%0A&pk1=a&pk2=d&pk3=v

select * from compound_three_primary_keys where (pk1 > :pk1) or (pk1 = :pk1 and pk2 > :pk2) or (pk1 = :pk1 and pk2 = :pk2 and pk3 > :pk3) order by pk1, pk2, pk3;

Here's how I generated the table for testing this with 3 compound primary keys:

CREATE_SQL = '''
CREATE TABLE compound_three_primary_keys (
    pk1 varchar(30),
    pk2 varchar(30),
    pk3 varchar(30),
    content text,
    PRIMARY KEY (pk1, pk2, pk3)
);'''
alphabet = 'abcdefghijklmnopqrstuvwxyz'
for a in alphabet:
    for b in alphabet:
        for c in alphabet:
            print('''
INSERT INTO compound_three_primary_keys VALUES ('{}', '{}', '{}', '{}');
        '''.strip().format(a, b, c, '{}-{}-{}-{}-{}-{}'.format(a,b,c,a,b,c)))

Without row values syntax, the necessary SQL to retrieve the next page after d, v gets a bit gnarly:

select * from compound_primary_key
where pk1 >= "d" and not (pk1 = "d" and pk2 <= "v")
order by pk1, pk2

See https://datasette-issue-190-compound-pks.now.sh/compound-pks-9aafe8f?sql=select+*+from+compound_primary_key+where+pk1+%3E%3D+%22d%22+and+not+%28pk1+%3D+%22d%22+and+pk2+%3C%3D+%22v%22%29+order+by+pk1%2C+pk2

This article was useful for figuring this out: https://use-the-index-luke.com/sql/partial-results/fetch-next-page

This is because the SQL we are using here is:

select * from compound_primary_key where "pk1" > "d" and "pk2" > "v" order by pk1, pk2 limit 101

This is incorrect. The correct SQL syntax (according to the example on https://www.sqlite.org/rowvalue.html#scrolling_window_queries ) is:

select * from compound_primary_key where ("pk1", "pk2") > ("d", "v") order by pk1, pk2 limit 101

BUT... this uses "row values" syntax which was only added to SQLite in version 3.15.0 in October 2016: https://sqlite.org/changes.html#version_3_15_0

The version on https://datasette-issue-190-compound-pks.now.sh/compound-pks-9aafe8f?sql=select+sqlite_version%28%29%3B is 3.8.7.1 from October 2014.

There is one other interesting option for auto-enabling/disabling sort: the inspect command could include data about column index presence and whether or not a column has any null values in it.

This would allow us to dynamically include a "nulls last" option but only for columns that contain at least one null.

It's quite a lot of additional engineering for a very minor feature though, so I think I'll punt on that for the moment.

We may find that the _group_count feature can benefit from column value statistics later on though.

I'm tempted to put these verbose sorting options inline in the page HTML but have them in the table footer so they don't clog up the top half of the page with uninteresting links - then use JavaScript to hoik them out into a dropdown menu attached to each column header.

I think there are actually four kinds of sort order we need to support;

• ascending
• descending
• ascending, nulls last
• descending, nulls last

It looks like [-blah] is a valid SQLite table name, so mark I descending with a hyphen prefix isn't good.

Instead, maybe this:

?_sort_asc=col1&_sort_desc_nulls_last=col2

In terms of user interface: the obvious place to put this is as a drop down menu on the column headers.

This also means the UI can support combined sort orders. Assuming you are already sorted by county descending and you select the candidate column header, the options could be:

• sort all by candidate
• sort all by candidate, descending
• sort by county descending, then by candidate
• sort by county descending, then by candidate descending

I'd like to continue to support _next=token pagination even for custom sort orders.

To do that I should include rowid (or general primary key) as the tie breaker on all sorts so I can incorporate that it into the _next= token.

This is a better pattern as you don't have to pick a minimum value:

ORDER BY CASE WHEN SOMECOL IS NULL THEN 1 ELSE 0 END, SOMECOL

Might have to do something special to get sort-by-nulls-last: https://stackoverflow.com/questions/12503120/how-to-do-nulls-last-in-sqlite

order by ifnull(column_name, -999999)

Would need to figure out a smart way to get the default value - maybe by running a min() or max() against the column first?

I think this can work with a ?_sort=xxx parameter - and ?_sort=-xxx to sort in the opposite direction.

I'd like to support "sort by X descending, then by Y ascending if there are dupes for X" as well. Two ways that could work:

?_sort=-xxx,yyy

Or...

?_sort=-xxx&_sort=yyy

The second option is probably better in that it makes it easier for columns to have a comma in their name.

Is it possible for a SQLite column to start with a - character?

Given how unlikely it is that this will pose a real problem I think I like option 1: enable sort-by-column by default for all tables, then allow power users to instead switch to explicit enabling of the functionality in their metadata.json if they know their data is too big.