issue_comments

15 rows where issue = 314665147 sorted by updated_at descending

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

user

issue

  • Bug: Sort by column with NULL in next_page URL · 15

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
381803157 https://github.com/simonw/datasette/issues/216#issuecomment-381803157 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTgwMzE1Nw== simonw 9599 2018-04-17T01:45:24Z 2018-04-17T01:45:24Z OWNER

Fixed!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381801302 https://github.com/simonw/datasette/issues/216#issuecomment-381801302 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTgwMTMwMg== simonw 9599 2018-04-17T01:33:43Z 2018-04-17T01:33:43Z OWNER

This is the SQL that returns differing results in production and on my laptop: https://datasette-issue-216-pagination.now.sh/sortable-5679797?sql=select+%2A+from+sortable+where+%28sortable_with_nulls+is+null+and+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29+order+by+sortable_with_nulls+desc+limit+51&p0=b&p1=t

select * from sortable where (sortable_with_nulls is null and ((pk1 > :p0)
  or
(pk1 = :p0 and pk2 > :p1))) order by sortable_with_nulls desc limit 51

I think that order by sortable_with_nulls desc bit is at fault - the primary keys should be included in that order by as well.

Sure enough, changing the query to this one returns the same results across both environments:

select * from sortable where (sortable_with_nulls is null and ((pk1 > :p0)
  or
(pk1 = :p0 and pk2 > :p1))) order by sortable_with_nulls desc, pk1, pk2 limit 51
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381799408 https://github.com/simonw/datasette/issues/216#issuecomment-381799408 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTc5OTQwOA== simonw 9599 2018-04-17T01:22:30Z 2018-04-17T01:22:30Z OWNER

... which is VERY surprising, because 3.23.0 only came out on 2nd April this year: https://www.sqlite.org/changes.html - I have no idea how I came to be running that version on my laptop.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381799267 https://github.com/simonw/datasette/issues/216#issuecomment-381799267 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTc5OTI2Nw== simonw 9599 2018-04-17T01:21:35Z 2018-04-17T01:21:35Z OWNER

The version that I deployed which exhibits the bug is running SQLite 3.8.7.1 - https://datasette-issue-216-pagination.now.sh/sortable-5679797?sql=select+sqlite_version%28%29

The version that I have running locally which does NOT exhibit the bug is running SQLite 3.23.0

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381798786 https://github.com/simonw/datasette/issues/216#issuecomment-381798786 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTc5ODc4Ng== simonw 9599 2018-04-17T01:18:25Z 2018-04-17T01:18:25Z OWNER

Here's the test that's failing:

https://github.com/simonw/datasette/blob/59a3aa859c0e782aeda9a515b1b52c358e8458a2/tests/test_api.py#L437-L470

I got Travis to spit out the fetched and expected variables.

expected has 201 items in it and is identical to what I get on my local laptop.

fetched has 250 items in it, so it's clearly different from my local environment.

I've managed to replicate the bug in production! I created a test database like this:

python tests/fixtures.py sortable.db

Then deployed that database like so:

datasette publish now sortable.db \
    --extra-options="--page_size=50" --branch=debug-travis-issue-216

And... if you click "next" on this page https://datasette-issue-216-pagination.now.sh/sortable-5679797/sortable?_sort_desc=sortable_with_nulls five times you get back 250 results, when you should only get back 201.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381794744 https://github.com/simonw/datasette/issues/216#issuecomment-381794744 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTc5NDc0NA== simonw 9599 2018-04-17T00:51:41Z 2018-04-17T00:51:41Z OWNER

I'm reverting this out of master until I can figure out why the tests are failing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381788051 https://github.com/simonw/datasette/issues/216#issuecomment-381788051 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTc4ODA1MQ== simonw 9599 2018-04-17T00:07:48Z 2018-04-17T00:07:48Z OWNER

Still failing. This is very odd.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381786522 https://github.com/simonw/datasette/issues/216#issuecomment-381786522 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTc4NjUyMg== simonw 9599 2018-04-16T23:58:45Z 2018-04-16T23:59:13Z OWNER

Weird... tests are failing in Travis, despite passing on my local machine. https://travis-ci.org/simonw/datasette/builds/367423706

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381649437 https://github.com/simonw/datasette/issues/216#issuecomment-381649437 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTY0OTQzNw== simonw 9599 2018-04-16T15:39:21Z 2018-04-16T15:39:21Z OWNER

Here's where that SQL gets constructed at the moment:

https://github.com/simonw/datasette/blob/10a34f995c70daa37a8a2aa02c3135a4b023a24c/datasette/app.py#L761-L771

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381649140 https://github.com/simonw/datasette/issues/216#issuecomment-381649140 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTY0OTE0MA== simonw 9599 2018-04-16T15:38:29Z 2018-04-16T15:38:29Z OWNER

But what would that SQL look like for _sort_desc?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381648053 https://github.com/simonw/datasette/issues/216#issuecomment-381648053 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTY0ODA1Mw== simonw 9599 2018-04-16T15:35:17Z 2018-04-16T15:35:17Z OWNER

I think the correct SQL is this: https://datasette-issue-189-demo-3.now.sh/salaries-7859114-7859114?sql=select+rowid%2C+*+from+%5B2017+Maryland+state+salaries%5D%0D%0Awhere+%28middle_initial+is+not+null+or+%28middle_initial+is+null+and+rowid+%3E+%3Ap0%29%29%0D%0Aorder+by+middle_initial+limit+101&p0=391

select rowid, * from [2017 Maryland state salaries]
where (middle_initial is not null or (middle_initial is null and rowid > :p0))
order by middle_initial limit 101

Though this will also need to be taken into account for #198

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381645973 https://github.com/simonw/datasette/issues/216#issuecomment-381645973 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTY0NTk3Mw== simonw 9599 2018-04-16T15:29:11Z 2018-04-16T15:29:11Z OWNER

I could use $null as a magic value that means None. Since I'm applying quote_plus() to actual values, any legit strings that look like this will be encoded as %24null:

>>> urllib.parse.quote_plus('$null')
'%24null'
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381645274 https://github.com/simonw/datasette/issues/216#issuecomment-381645274 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTY0NTI3NA== simonw 9599 2018-04-16T15:27:16Z 2018-04-16T15:27:16Z OWNER

Relevant code:

https://github.com/simonw/datasette/blob/904f1c75a3c17671d25c53b91e177c249d14ab3b/datasette/app.py#L828-L832

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381644355 https://github.com/simonw/datasette/issues/216#issuecomment-381644355 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTY0NDM1NQ== simonw 9599 2018-04-16T15:24:38Z 2018-04-16T15:24:38Z OWNER

So there are two tricky problems to solve here:

  • I need a way of encoding null into that _next= that is unambiguous from the string None or null. This means introducing some kind of escaping mechanism in those strings. I already use URL encoding as part of the construction of those components here, maybe that can help here?
  • I need to figure out what the SQL should be for the "next" set of results if the previous value was null. Thankfully we use the primary key as a tie-breaker so this shouldn't be impossible.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147
381643173 https://github.com/simonw/datasette/issues/216#issuecomment-381643173 https://api.github.com/repos/simonw/datasette/issues/216 MDEyOklzc3VlQ29tbWVudDM4MTY0MzE3Mw== simonw 9599 2018-04-16T15:21:17Z 2018-04-16T15:21:17Z OWNER

Yikes, definitely a bug.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug: Sort by column with NULL in next_page URL 314665147

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 33.77ms · About: github-to-sqlite