issue_comments

Table actions

8 rows where issue = 325958506 and "updated_at" is on date 2021-02-18 sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 2

author_association 2

issue 1

  • Support cross-database joins · 8 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
781670827 https://github.com/simonw/datasette/issues/283#issuecomment-781670827 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTY3MDgyNw== simonw 9599 2021-02-18T22:16:46Z 2021-02-18T22:16:46Z OWNER

Demo is now live here: https://latest.datasette.io/_memory

The documentation is at https://docs.datasette.io/en/latest/sql_queries.html#cross-database-queries - it links to this example query: https://latest.datasette.io/_memory?sql=select%0D%0A++%27fixtures%27+as+database%2C+%0D%0Afrom%0D%0A++%5Bfixtures%5D.sqlite_master%0D%0Aunion%0D%0Aselect%0D%0A++%27extra_database%27+as+database%2C+%0D%0Afrom%0D%0A++%5Bextra_database%5D.sqlite_master

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support cross-database joins 325958506  
781665560 https://github.com/simonw/datasette/issues/283#issuecomment-781665560 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTY2NTU2MA== simonw 9599 2021-02-18T22:06:14Z 2021-02-18T22:06:14Z OWNER

The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run datasette one.db two.db three.db --crossdb and then use the /_memory page to run joins across tables from multiple databases.

It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see Datasette Library) this won't be particularly useful for you.

So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting /_memory?attach=db1&attach=db2 to get a special connection.

Also worth noting: plugins that implement the prepare_connection() hook can attach additional databases - so if you need better, customized support for this one way to handle that would be with a custom plugin.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support cross-database joins 325958506  
781593169 https://github.com/simonw/datasette/issues/283#issuecomment-781593169 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTU5MzE2OQ== simonw 9599 2021-02-18T19:47:34Z 2021-02-18T19:47:34Z OWNER

I have a working version now, moving development to a pull request.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support cross-database joins 325958506  
781591015 https://github.com/simonw/datasette/issues/283#issuecomment-781591015 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTU5MTAxNQ== simonw 9599 2021-02-18T19:44:02Z 2021-02-18T19:44:02Z OWNER

For the moment I'm going to hard-code a SQLITE_LIMIT_ATTACHED=10 constant and only attach the first 10 databases to the _memory connection.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support cross-database joins 325958506  
781574786 https://github.com/simonw/datasette/issues/283#issuecomment-781574786 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTU3NDc4Ng== simonw 9599 2021-02-18T19:15:37Z 2021-02-18T19:15:37Z OWNER

select * from pragma_database_list(); is useful - shows all attached databases for the current connection.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support cross-database joins 325958506  
781573676 https://github.com/simonw/datasette/issues/283#issuecomment-781573676 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTU3MzY3Ng== simonw 9599 2021-02-18T19:13:30Z 2021-02-18T19:13:30Z OWNER

It turns out SQLite defaults to a maximum of 10 attached databases. This can be increased using a compile-time constant, but even with that it cannot be more than 62: https://stackoverflow.com/questions/9845448/attach-limit-10

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support cross-database joins 325958506  
781077127 https://github.com/simonw/datasette/issues/283#issuecomment-781077127 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTA3NzEyNw== simonw 9599 2021-02-18T05:56:30Z 2021-02-18T05:57:34Z OWNER

I'm going to to try prototyping the --crossdb option that causes /_memory to connect to all databases as a starting point and see how well that works.

 
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 1,
    "eyes": 0
}
 Support cross-database joins 325958506  
780991910 https://github.com/simonw/datasette/issues/283#issuecomment-780991910 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MDk5MTkxMA== rayvoelker 9308268 2021-02-18T02:13:56Z 2021-02-18T02:13:56Z NONE

I was going ask you about this issue when we talk during your office-hours schedule this Friday, but was there any support ever added for doing this cross-database joining?

I have a use-case where could be pretty neat to do analysis using this tool on time-specific databases from snapshots

https://ilsweb.cincinnatilibrary.org/collection-analysis/

and thanks again for such an amazing tool!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support cross-database joins 325958506  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);