issue_comments

Table actions

4 rows where author_association = "CONTRIBUTOR" and issue = 1560982210 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

issue 1

  • array facet: don't materialize unnecessary columns · 4 ✖

author_association 1

  • CONTRIBUTOR · 4 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1407716963 https://github.com/simonw/datasette/pull/2008#issuecomment-1407716963 https://api.github.com/repos/simonw/datasette/issues/2008 IC_kwDOBm6k_c5T6A5j cldellow 193185 2023-01-29T17:04:03Z 2023-01-29T17:04:03Z CONTRIBUTOR

Performance tests - I think most places don't have them as a formal gate enforced by CI. TypeScript and scalac seem to have tests that run to capture timings. The timings are included by a bot as a comment or build check, and also stored in a database so you can graph changes over time to spot regressions. Probably overkill for Datasette!

Window functions - oh, good point. Looks like Ubuntu shipped JSON1 support as far back as sqlite 3.11. I'll let this PR linger until there's a way to run against different SQLite versions. For now, I'm shipping this with datasette-ui-extras, since I think it's OK for a plugin to enforce a higher minimum requirement.

Tests - there actually did end up being test changes to capture the undercount bug of the current implementation, so the current implementation would fail against the new tests.

Perhaps a non-window function version could be written that uses random() instead of row_number() over () in order to get a unique key. It's technically not unique, but in practice, I imagine it'll work well.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 array facet: don't materialize unnecessary columns 1560982210  
1407561308 https://github.com/simonw/datasette/pull/2008#issuecomment-1407561308 https://api.github.com/repos/simonw/datasette/issues/2008 IC_kwDOBm6k_c5T5a5c cldellow 193185 2023-01-29T04:50:50Z 2023-01-29T04:50:50Z CONTRIBUTOR

I pushed a revised version which ends up being faster -- the example which currently takes 4 seconds now runs in 500ms.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 array facet: don't materialize unnecessary columns 1560982210  
1407558284 https://github.com/simonw/datasette/pull/2008#issuecomment-1407558284 https://api.github.com/repos/simonw/datasette/issues/2008 IC_kwDOBm6k_c5T5aKM cldellow 193185 2023-01-29T04:23:58Z 2023-01-29T04:24:27Z CONTRIBUTOR

Ack, this PR is broken. I see now that the inner.* is necessary for ensuring the correct count in the face of rows having duplicate values in views.

That fixes the overcounting, but I think can undercount when the rows have the same data, eg a view like:

sql SELECT '["bar"]' tags UNION ALL SELECT '["bar"]'

will produce a count of {"bar": 1 }, when it should be {"bar": 2}. In fact, this could apply in tables without primary keys, too.

If inner came from a base table that had a primary key or a rowid, we could use those column(s) to solve that case.

I guess a general solution would be to compute a window function so we have a distinct ID for each row. Will fiddle to see if I can get that working.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 array facet: don't materialize unnecessary columns 1560982210  
1407470429 https://github.com/simonw/datasette/pull/2008#issuecomment-1407470429 https://api.github.com/repos/simonw/datasette/issues/2008 IC_kwDOBm6k_c5T5Etd cldellow 193185 2023-01-28T19:34:29Z 2023-01-28T19:34:29Z CONTRIBUTOR

I don't know how/if you do automated tests for performance, so I haven't changed any of the tests.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 array facet: don't materialize unnecessary columns 1560982210  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);