issue_comments

Table actions

2 rows where author_association = "OWNER" and issue = 1433576351 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

issue 1

  • Datasette with many and large databases > Memory use · 2 ✖

author_association 1

  • OWNER · 2 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1311273063 https://github.com/simonw/datasette/issues/1880#issuecomment-1311273063 https://api.github.com/repos/simonw/datasette/issues/1880 IC_kwDOBm6k_c5OKHBn simonw 9599 2022-11-11T06:15:28Z 2022-11-11T06:15:28Z OWNER

The _internal database is intended to help Datasette handle much larger attached databases. Right now Datasette attempts to show every database on the https://latest.datasette.io/ index page and every table on the https://latest.datasette.io/fixtures database index page - but these are not paginated. If you had a database containing 1,000 tables the database index page would get pretty slow.

So I want to be able to paginate (and search) those. But to paginate them it's useful to have them in a database table itself, since then I can paginate using SQL.

My plan for _internal is to use it to implement those advanced browsing features. I've not completed this work yet though. See this issue for more details on that:

  • 417

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Datasette with many and large databases > Memory use 1433576351  
1311271298 https://github.com/simonw/datasette/issues/1880#issuecomment-1311271298 https://api.github.com/repos/simonw/datasette/issues/1880 IC_kwDOBm6k_c5OKGmC simonw 9599 2022-11-11T06:12:29Z 2022-11-11T06:12:29Z OWNER

I think you may have misunderstood this feature. This is talking about the _internal in-memory database, which maintains a set of tables that list the databases and tables that are attached to Datasette.

They're not a copy of the data itself - just a list of table names, column names and database names.

You can see what that database looks like by signing in as root - running datasette --root and clicking the link. Or you can see an example here:

For the example instance that looks like this:

The two most interesting tables in there are these ones:

As you can see, it's just the table schema itself and the columns that make up the tables. Even if you have hundreds of databases connected each with hundreds of tables this should still only add up to a few MB of RAM.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Datasette with many and large databases > Memory use 1433576351  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);