issue_comments

Table actions

5 rows where issue = 333238932 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 2

author_association 2

issue 1

  • datasette inspect takes a very long time on large dbs · 5 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
491533700 https://github.com/simonw/datasette/issues/316#issuecomment-491533700 https://api.github.com/repos/simonw/datasette/issues/316 MDEyOklzc3VlQ29tbWVudDQ5MTUzMzcwMA== simonw 9599 2019-05-11T18:26:25Z 2019-05-11T18:26:25Z OWNER

This will be fixed by #419 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette inspect takes a very long time on large dbs 333238932  
398133159 https://github.com/simonw/datasette/issues/316#issuecomment-398133159 https://api.github.com/repos/simonw/datasette/issues/316 MDEyOklzc3VlQ29tbWVudDM5ODEzMzE1OQ== simonw 9599 2018-06-18T17:29:59Z 2018-07-10T15:14:53Z OWNER

For #271 I've been contemplating having Datasette work against an on-disk database that gets modified without needing to restart the server.

For that to work, I'll have to dramatically change the inspect() mechanism. It may be that inspect becomes an optional optimization in the future.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette inspect takes a very long time on large dbs 333238932  
398109204 https://github.com/simonw/datasette/issues/316#issuecomment-398109204 https://api.github.com/repos/simonw/datasette/issues/316 MDEyOklzc3VlQ29tbWVudDM5ODEwOTIwNA== gavinband 132230 2018-06-18T16:12:45Z 2018-06-18T16:12:45Z NONE

Hi Simon, Thanks for the response. Ok I'll try running datasette inspect up front. In principle the db won't change. However, the site's in development and it's likely I'll need to add views and some auxiliary (smaller) tables as I go along. I will need to be careful with this if it involves an inspect step in each iteration, though. g.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette inspect takes a very long time on large dbs 333238932  
398101670 https://github.com/simonw/datasette/issues/316#issuecomment-398101670 https://api.github.com/repos/simonw/datasette/issues/316 MDEyOklzc3VlQ29tbWVudDM5ODEwMTY3MA== simonw 9599 2018-06-18T15:49:35Z 2018-06-18T15:50:38Z OWNER

Wow, I've gone as high as 7GB but I've never tried it against 600GB.

datasette inspect is indeed expected to take a long time for large databases. That's why it's available as a separate command: by running datasette inspect to generate inspect-data.json you can execute it just once against a large database and then have datasette serve take advantage of that cached metadata (hence avoiding datasette serve hanging on startup).

As you spotted, most of the time is spent in those counts. I imagine you don't need those row counts in order for the rest of Datasette to function correctly (they are mainly used for display purposes - on the https://latest.datasette.io/fixtures index page for example).

If your database changes infrequently, for the moment I recommend running datasette inspect once to generate the inspect-data.json file (let me know how long it takes) and then passing that file to datasette serve mydb.db --inspect-file=inspect-data.json

If your database DOES change frequently then this workaround won't help you much. Let me know and I'll see how much work it would take to have those row counts be optional rather than required.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette inspect takes a very long time on large dbs 333238932  
398030903 https://github.com/simonw/datasette/issues/316#issuecomment-398030903 https://api.github.com/repos/simonw/datasette/issues/316 MDEyOklzc3VlQ29tbWVudDM5ODAzMDkwMw== gavinband 132230 2018-06-18T12:00:43Z 2018-06-18T12:00:43Z NONE

I should add that I'm using datasette version 0.22, Python 2.7.10 on Mac OS X. Happy to send more info if helpful.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette inspect takes a very long time on large dbs 333238932  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);