{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381803157", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381803157, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTgwMzE1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-17T01:45:24Z", "updated_at": "2018-04-17T01:45:24Z", "author_association": "OWNER", "body": "Fixed!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381801302", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381801302, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTgwMTMwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-17T01:33:43Z", "updated_at": "2018-04-17T01:33:43Z", "author_association": "OWNER", "body": "This is the SQL that returns differing results in production and on my laptop: https://datasette-issue-216-pagination.now.sh/sortable-5679797?sql=select+%2A+from+sortable+where+%28sortable_with_nulls+is+null+and+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29+order+by+sortable_with_nulls+desc+limit+51&p0=b&p1=t\r\n\r\n```\r\nselect * from sortable where (sortable_with_nulls is null and ((pk1 > :p0)\r\n  or\r\n(pk1 = :p0 and pk2 > :p1))) order by sortable_with_nulls desc limit 51\r\n```\r\nI think that `order by sortable_with_nulls desc` bit is at fault - the primary keys should be included in that order by as well.\r\n\r\nSure enough, changing the query to this one returns the same results across both environments:\r\n\r\n```\r\nselect * from sortable where (sortable_with_nulls is null and ((pk1 > :p0)\r\n  or\r\n(pk1 = :p0 and pk2 > :p1))) order by sortable_with_nulls desc, pk1, pk2 limit 51\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381799408", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381799408, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTc5OTQwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-17T01:22:30Z", "updated_at": "2018-04-17T01:22:30Z", "author_association": "OWNER", "body": "... which is VERY surprising, because `3.23.0` only came out on 2nd April this year: https://www.sqlite.org/changes.html - I have no idea how I came to be running that version on my laptop.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381799267", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381799267, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTc5OTI2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-17T01:21:35Z", "updated_at": "2018-04-17T01:21:35Z", "author_association": "OWNER", "body": "The version that I deployed which exhibits the bug is running SQLite `3.8.7.1` - https://datasette-issue-216-pagination.now.sh/sortable-5679797?sql=select+sqlite_version%28%29\r\n\r\nThe version that I have running locally which does NOT exhibit the bug is running SQLite `3.23.0`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381798786", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381798786, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTc5ODc4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-17T01:18:25Z", "updated_at": "2018-04-17T01:18:25Z", "author_association": "OWNER", "body": "Here's the test that's failing:\r\n\r\nhttps://github.com/simonw/datasette/blob/59a3aa859c0e782aeda9a515b1b52c358e8458a2/tests/test_api.py#L437-L470\r\n\r\nI got Travis to spit out the `fetched` and `expected` variables.\r\n\r\n`expected` has 201 items in it and is identical to what I get on my local laptop.\r\n\r\n`fetched` has 250 items in it, so it's clearly different from my local environment.\r\n\r\nI've managed to replicate the bug in production! I created a test database like this:\r\n\r\n    python tests/fixtures.py sortable.db\r\n\r\nThen deployed that database like so:\r\n\r\n    datasette publish now sortable.db \\\r\n        --extra-options=\"--page_size=50\" --branch=debug-travis-issue-216\r\n\r\nAnd... if you click \"next\" on this page https://datasette-issue-216-pagination.now.sh/sortable-5679797/sortable?_sort_desc=sortable_with_nulls five times you get back 250 results, when you should only get back 201.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381794744", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381794744, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTc5NDc0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-17T00:51:41Z", "updated_at": "2018-04-17T00:51:41Z", "author_association": "OWNER", "body": "I'm reverting this out of master until I can figure out why the tests are failing.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381788051", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381788051, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTc4ODA1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-17T00:07:48Z", "updated_at": "2018-04-17T00:07:48Z", "author_association": "OWNER", "body": "Still failing. This is very odd.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381786522", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381786522, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTc4NjUyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-16T23:58:45Z", "updated_at": "2018-04-16T23:59:13Z", "author_association": "OWNER", "body": "Weird... tests are failing in Travis, despite passing on my local machine. https://travis-ci.org/simonw/datasette/builds/367423706", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381649437", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381649437, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTY0OTQzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-16T15:39:21Z", "updated_at": "2018-04-16T15:39:21Z", "author_association": "OWNER", "body": "Here's where that SQL gets constructed at the moment:\r\n\r\nhttps://github.com/simonw/datasette/blob/10a34f995c70daa37a8a2aa02c3135a4b023a24c/datasette/app.py#L761-L771", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381649140", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381649140, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTY0OTE0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-16T15:38:29Z", "updated_at": "2018-04-16T15:38:29Z", "author_association": "OWNER", "body": "But what would that SQL look like for `_sort_desc`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381648053", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381648053, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTY0ODA1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-16T15:35:17Z", "updated_at": "2018-04-16T15:35:17Z", "author_association": "OWNER", "body": "I think the correct SQL is this: https://datasette-issue-189-demo-3.now.sh/salaries-7859114-7859114?sql=select+rowid%2C+*+from+%5B2017+Maryland+state+salaries%5D%0D%0Awhere+%28middle_initial+is+not+null+or+%28middle_initial+is+null+and+rowid+%3E+%3Ap0%29%29%0D%0Aorder+by+middle_initial+limit+101&p0=391\r\n\r\n```\r\nselect rowid, * from [2017 Maryland state salaries]\r\nwhere (middle_initial is not null or (middle_initial is null and rowid > :p0))\r\norder by middle_initial limit 101\r\n```\r\n\r\nThough this will also need to be taken into account for #198 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381645973", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381645973, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTY0NTk3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-16T15:29:11Z", "updated_at": "2018-04-16T15:29:11Z", "author_association": "OWNER", "body": "I could use `$null` as a magic value that means None. Since I'm applying `quote_plus()` to actual values, any legit strings that look like this will be encoded as `%24null`:\r\n\r\n```\r\n>>> urllib.parse.quote_plus('$null')\r\n'%24null'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381645274", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381645274, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTY0NTI3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-16T15:27:16Z", "updated_at": "2018-04-16T15:27:16Z", "author_association": "OWNER", "body": "Relevant code:\r\n\r\nhttps://github.com/simonw/datasette/blob/904f1c75a3c17671d25c53b91e177c249d14ab3b/datasette/app.py#L828-L832", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381644355", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381644355, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTY0NDM1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-16T15:24:38Z", "updated_at": "2018-04-16T15:24:38Z", "author_association": "OWNER", "body": "So there are two tricky problems to solve here:\r\n\r\n* I need a way of encoding `null` into that `_next=` that is unambiguous from the string `None` or `null`. This means introducing some kind of escaping mechanism in those strings. I already use URL encoding as part of the construction of those components here, maybe that can help here?\r\n* I need to figure out what the SQL should be for the \"next\" set of results if the previous value was null. Thankfully we use the primary key as a tie-breaker so this shouldn't be impossible.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/216#issuecomment-381643173", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/216", "id": 381643173, "node_id": "MDEyOklzc3VlQ29tbWVudDM4MTY0MzE3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-04-16T15:21:17Z", "updated_at": "2018-04-16T15:21:17Z", "author_association": "OWNER", "body": "Yikes, definitely a bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 314665147, "label": "Bug: Sort by column with NULL in next_page URL"}, "performed_via_github_app": null}