issue_comments

Table actions

3 rows where issue = 751195017 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 2

author_association 2

issue 1

  • Accessing a database's `.json` is slow for very large SQLite files · 3 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
754184287 https://github.com/simonw/datasette/issues/1111#issuecomment-754184287 https://api.github.com/repos/simonw/datasette/issues/1111 MDEyOklzc3VlQ29tbWVudDc1NDE4NDI4Nw== simonw 9599 2021-01-04T19:57:53Z 2021-01-04T19:57:53Z OWNER

Relevant new feature in sqlite-utils: the ability to use triggers to maintain fast counts. This optimization could help a lot here. https://github.com/simonw/sqlite-utils/issues/212

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Accessing a database's `.json` is slow for very large SQLite files 751195017  
736322290 https://github.com/simonw/datasette/issues/1111#issuecomment-736322290 https://api.github.com/repos/simonw/datasette/issues/1111 MDEyOklzc3VlQ29tbWVudDczNjMyMjI5MA== abdusco 3243482 2020-12-01T08:54:47Z 2020-12-01T08:54:47Z CONTRIBUTOR

Somewhat related: https://github.com/simonw/datasette/issues/859 I fixed the issue with forking and disabling the counts for hidden tables.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Accessing a database's `.json` is slow for very large SQLite files 751195017  
735320736 https://github.com/simonw/datasette/issues/1111#issuecomment-735320736 https://api.github.com/repos/simonw/datasette/issues/1111 MDEyOklzc3VlQ29tbWVudDczNTMyMDczNg== simonw 9599 2020-11-29T02:46:23Z 2020-11-29T02:46:23Z OWNER

This is a really useful bug report, thanks! I agree: more aggressive timeouts on table counts sounds like the right solution here.

I've learned that avoiding count(*) is crucial for handling these larger databases. Datasette has been getting better about that over time but there are still some edge-cases.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Accessing a database's `.json` is slow for very large SQLite files 751195017  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);