issue_comments

Table actions

3 rows where issue = 333238932 and user = 9599 sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • simonw · 3 ✖

issue 1

  • datasette inspect takes a very long time on large dbs · 3 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
491533700 https://github.com/simonw/datasette/issues/316#issuecomment-491533700 https://api.github.com/repos/simonw/datasette/issues/316 MDEyOklzc3VlQ29tbWVudDQ5MTUzMzcwMA== simonw 9599 2019-05-11T18:26:25Z 2019-05-11T18:26:25Z OWNER

This will be fixed by #419 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette inspect takes a very long time on large dbs 333238932  
398133159 https://github.com/simonw/datasette/issues/316#issuecomment-398133159 https://api.github.com/repos/simonw/datasette/issues/316 MDEyOklzc3VlQ29tbWVudDM5ODEzMzE1OQ== simonw 9599 2018-06-18T17:29:59Z 2018-07-10T15:14:53Z OWNER

For #271 I've been contemplating having Datasette work against an on-disk database that gets modified without needing to restart the server.

For that to work, I'll have to dramatically change the inspect() mechanism. It may be that inspect becomes an optional optimization in the future.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette inspect takes a very long time on large dbs 333238932  
398101670 https://github.com/simonw/datasette/issues/316#issuecomment-398101670 https://api.github.com/repos/simonw/datasette/issues/316 MDEyOklzc3VlQ29tbWVudDM5ODEwMTY3MA== simonw 9599 2018-06-18T15:49:35Z 2018-06-18T15:50:38Z OWNER

Wow, I've gone as high as 7GB but I've never tried it against 600GB.

datasette inspect is indeed expected to take a long time for large databases. That's why it's available as a separate command: by running datasette inspect to generate inspect-data.json you can execute it just once against a large database and then have datasette serve take advantage of that cached metadata (hence avoiding datasette serve hanging on startup).

As you spotted, most of the time is spent in those counts. I imagine you don't need those row counts in order for the rest of Datasette to function correctly (they are mainly used for display purposes - on the https://latest.datasette.io/fixtures index page for example).

If your database changes infrequently, for the moment I recommend running datasette inspect once to generate the inspect-data.json file (let me know how long it takes) and then passing that file to datasette serve mydb.db --inspect-file=inspect-data.json

If your database DOES change frequently then this workaround won't help you much. Let me know and I'll see how much work it would take to have those row counts be optional rather than required.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette inspect takes a very long time on large dbs 333238932  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);