issue_comments

Table actions

4 rows where issue = 1433576351 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 2

author_association 2

issue 1

  • Datasette with many and large databases > Memory use · 4 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1317420812 https://github.com/simonw/datasette/issues/1880#issuecomment-1317420812 https://api.github.com/repos/simonw/datasette/issues/1880 IC_kwDOBm6k_c5Ohj8M amitkoth 525934 2022-11-16T17:50:29Z 2022-11-16T17:50:29Z NONE

I appreciate your response @simonw - thanks!

I'll clarify what we need further - let's imagine we have 2000 SQLLite databases (for 2000 tenants), but we only want to run one datasette instance for each of those tenants to query/use datasette against their own database only. This means the "connection" between datasette and the SQLLite database would be dynamic, based on the tenantID that's required on an incoming request.

Is there any specific config or other considerations in this use case, to minimize memory use on a single, efficient VM and serve queries to all these tenants?

cc @muadham

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Datasette with many and large databases > Memory use 1433576351  
1311273063 https://github.com/simonw/datasette/issues/1880#issuecomment-1311273063 https://api.github.com/repos/simonw/datasette/issues/1880 IC_kwDOBm6k_c5OKHBn simonw 9599 2022-11-11T06:15:28Z 2022-11-11T06:15:28Z OWNER

The _internal database is intended to help Datasette handle much larger attached databases. Right now Datasette attempts to show every database on the https://latest.datasette.io/ index page and every table on the https://latest.datasette.io/fixtures database index page - but these are not paginated. If you had a database containing 1,000 tables the database index page would get pretty slow.

So I want to be able to paginate (and search) those. But to paginate them it's useful to have them in a database table itself, since then I can paginate using SQL.

My plan for _internal is to use it to implement those advanced browsing features. I've not completed this work yet though. See this issue for more details on that:

  • 417

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Datasette with many and large databases > Memory use 1433576351  
1311271298 https://github.com/simonw/datasette/issues/1880#issuecomment-1311271298 https://api.github.com/repos/simonw/datasette/issues/1880 IC_kwDOBm6k_c5OKGmC simonw 9599 2022-11-11T06:12:29Z 2022-11-11T06:12:29Z OWNER

I think you may have misunderstood this feature. This is talking about the _internal in-memory database, which maintains a set of tables that list the databases and tables that are attached to Datasette.

They're not a copy of the data itself - just a list of table names, column names and database names.

You can see what that database looks like by signing in as root - running datasette --root and clicking the link. Or you can see an example here:

For the example instance that looks like this:

The two most interesting tables in there are these ones:

As you can see, it's just the table schema itself and the columns that make up the tables. Even if you have hundreds of databases connected each with hundreds of tables this should still only add up to a few MB of RAM.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Datasette with many and large databases > Memory use 1433576351  
1301043042 https://github.com/simonw/datasette/issues/1880#issuecomment-1301043042 https://api.github.com/repos/simonw/datasette/issues/1880 IC_kwDOBm6k_c5NjFdi amitkoth 525934 2022-11-02T18:20:14Z 2022-11-02T18:20:14Z NONE

Follow on question - is all memory use @simonw - for both datasette and SQLlite confined to the "query time" itself i.e. the memory use is relevant only to a particular transaction or query - and then subsequently released?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Datasette with many and large databases > Memory use 1433576351  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);