24 rows where issue = 325958506 sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date), updated_at (date)

user

author_association

issue

  • Support cross-database joins · 24
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
781764561 https://github.com/simonw/datasette/issues/283#issuecomment-781764561 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTc2NDU2MQ== simonw 9599 2021-02-19T02:10:21Z 2021-02-19T02:10:21Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
781670827 https://github.com/simonw/datasette/issues/283#issuecomment-781670827 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTY3MDgyNw== simonw 9599 2021-02-18T22:16:46Z 2021-02-18T22:16:46Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
781665560 https://github.com/simonw/datasette/issues/283#issuecomment-781665560 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTY2NTU2MA== simonw 9599 2021-02-18T22:06:14Z 2021-02-18T22:06:14Z OWNER

The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run datasette one.db two.db three.db --crossdb and then use the /_memory page to run joins across tables from multiple databases.

It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see Datasette Library) this won't be particularly useful for you.

So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting /_memory?attach=db1&attach=db2 to get a special connection.

Also worth noting: plugins that implement the prepare_connection() hook can attach additional databases - so if you need better, customized support for this one way to handle that would be with a custom plugin.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
781593169 https://github.com/simonw/datasette/issues/283#issuecomment-781593169 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTU5MzE2OQ== simonw 9599 2021-02-18T19:47:34Z 2021-02-18T19:47:34Z OWNER

I have a working version now, moving development to a pull request.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
781591015 https://github.com/simonw/datasette/issues/283#issuecomment-781591015 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTU5MTAxNQ== simonw 9599 2021-02-18T19:44:02Z 2021-02-18T19:44:02Z OWNER

For the moment I'm going to hard-code a SQLITE_LIMIT_ATTACHED=10 constant and only attach the first 10 databases to the _memory connection.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
781574786 https://github.com/simonw/datasette/issues/283#issuecomment-781574786 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTU3NDc4Ng== simonw 9599 2021-02-18T19:15:37Z 2021-02-18T19:15:37Z OWNER

select * from pragma_database_list(); is useful - shows all attached databases for the current connection.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
781573676 https://github.com/simonw/datasette/issues/283#issuecomment-781573676 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTU3MzY3Ng== simonw 9599 2021-02-18T19:13:30Z 2021-02-18T19:13:30Z OWNER

It turns out SQLite defaults to a maximum of 10 attached databases. This can be increased using a compile-time constant, but even with that it cannot be more than 62: https://stackoverflow.com/questions/9845448/attach-limit-10

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
781077127 https://github.com/simonw/datasette/issues/283#issuecomment-781077127 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MTA3NzEyNw== simonw 9599 2021-02-18T05:56:30Z 2021-02-18T05:57:34Z OWNER

I'm going to to try prototyping the --crossdb option that causes /_memory to connect to all databases as a starting point and see how well that works.

{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 1,
    "eyes": 0
}
Support cross-database joins 325958506  
780991910 https://github.com/simonw/datasette/issues/283#issuecomment-780991910 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDc4MDk5MTkxMA== rayvoelker 9308268 2021-02-18T02:13:56Z 2021-02-18T02:13:56Z NONE

I was going ask you about this issue when we talk during your office-hours schedule this Friday, but was there any support ever added for doing this cross-database joining?

I have a use-case where could be pretty neat to do analysis using this tool on time-specific databases from snapshots

https://ilsweb.cincinnatilibrary.org/collection-analysis/

and thanks again for such an amazing tool!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
552140975 https://github.com/simonw/datasette/issues/283#issuecomment-552140975 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDU1MjE0MDk3NQ== simonw 9599 2019-11-09T21:51:41Z 2019-11-09T21:51:41Z OWNER

It may turn out that we have to recommend NOT exposing a Datasette instance to the public with dozens of database files that has multi-db queries enabled - will need to load test to understand if this recommendation is needed or not.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
552140870 https://github.com/simonw/datasette/issues/283#issuecomment-552140870 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDU1MjE0MDg3MA== simonw 9599 2019-11-09T21:49:51Z 2019-11-09T21:49:51Z OWNER

Better idea: if you run Datasette in cross-database joining mode, all connections start out as memory connections and then have new databases attached to them on-demand.

All table view queries will be automatically rewritten to start SELECT db.table.one, db.table.two FROM db.table ...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
537716955 https://github.com/simonw/datasette/issues/283#issuecomment-537716955 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDUzNzcxNjk1NQ== simonw 9599 2019-10-02T23:02:15Z 2019-10-02T23:02:15Z OWNER

I've been thinking pretty hard about this as part of #569. My big concerns are:

  • If I'm caching and reusing connections I need to worry about the different combinations - if I have four databases do I cache separate connections for the ("one", "two") AND ("two", "three") AND ("one", "three") and so on pairs?
  • How does the API and interface deal with instances where you have a database connected as the primary and you want to ATTACH another database and talk to that as well?

I think the best way to do this is to say that cross-database joins will only be available against the :memory: database. Maybe with an optional mode you can run like datasette --crossdb which causes every database to be ATTACHd to that connection with an alias so you can start running queries.

If this proves to be a problem when hundreds of files are attached to a Datasette Library instance (#417) then maybe cross database joins are handled (in that case) by the authenticated user selecting which ones to ?_attach= and detaching them at the end of the request. Also perhaps limit to joining across a maximum of 3 databases at once in this case.

I can probably avoid the scariest negative consequences of cross-database joins by having them turned off by default for signed-out users. The datasette-on-my-laptop or authenticated Datasette Library cases can be opt-in and can be a little less locked down.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391768302 https://github.com/simonw/datasette/issues/283#issuecomment-391768302 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc2ODMwMg== simonw 9599 2018-05-24T16:00:05Z 2018-05-24T16:00:05Z OWNER

I like /-/all-5de27e3 for this (with /-/all redirecting to the correct hash)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391756841 https://github.com/simonw/datasette/issues/283#issuecomment-391756841 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1Njg0MQ== simonw 9599 2018-05-24T15:27:42Z 2018-05-24T15:27:42Z OWNER

For an example query that pre-populates that textarea... maybe a UNION that pulls the first 10 rows from the first table of each of the first two databases?

select * from (select rowid, actors from fivethirtyeight.[love-actually/love_actually_adjacencies] limit 10)
   union all
select * from (select rowid, city from [google-trends].[20150430_UKDebate] limit 10)

https://datasette-cross-database-joins-prototype.now.sh/memory?sql=select+*+from+%28select+rowid%2C+actors+from+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D+limit+10%29%0D%0A+++union+all%0D%0Aselect+*+from+%28select+rowid%2C+city+from+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D+limit+10%29

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391755300 https://github.com/simonw/datasette/issues/283#issuecomment-391755300 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1NTMwMA== simonw 9599 2018-05-24T15:23:37Z 2018-05-24T15:23:37Z OWNER

On the /-all-5de27e3 page we can show the regular https://fivethirtyeight.datasettes.com/fivethirtyeight-5de27e3 interface but instead of the list of tables we can show a list of attached databases plus some help text showing how to construct a cross-database join.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391754506 https://github.com/simonw/datasette/issues/283#issuecomment-391754506 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1NDUwNg== simonw 9599 2018-05-24T15:21:37Z 2018-05-24T15:21:53Z OWNER

Giving it /all/ would be easier since that way the existing URL routes (including canned queries) would all work... but I would have to teach it NOT to expect a database content hash on that URL.

Or maybe it should still have a content hash (to enable far-future cache expiry headers on query results) but the hash should be constructed out of all of the other database hashes concatenated together.

That way the URLs would be /all-5de27e3 and /all-5de27e3/canned-query-name

Only downside: this would make it impossible to have a database file with the name all.db. I think that's probably an OK trade-off. You could turn the feature off with a config flag if you really want to use that filename (for whatever reason).

How about /-all-5de27e3/ instead to avoid collisions?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391752882 https://github.com/simonw/datasette/issues/283#issuecomment-391752882 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1Mjg4Mg== simonw 9599 2018-05-24T15:17:10Z 2018-05-24T15:17:10Z OWNER

Another option: give this the /-/all URL namespace.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391752629 https://github.com/simonw/datasette/issues/283#issuecomment-391752629 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1MjYyOQ== simonw 9599 2018-05-24T15:16:25Z 2018-05-24T15:16:25Z OWNER

Should this support canned queries too? I think it should, though that raises interesting questions regarding their URL structure.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391752425 https://github.com/simonw/datasette/issues/283#issuecomment-391752425 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1MjQyNQ== simonw 9599 2018-05-24T15:15:51Z 2018-05-24T15:15:51Z OWNER

This would make Datasett's SQL features a lot more instantly obvious to people who land on a homepage, which is probably a good thing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391752218 https://github.com/simonw/datasette/issues/283#issuecomment-391752218 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1MjIxOA== simonw 9599 2018-05-24T15:15:19Z 2018-05-24T15:15:19Z OWNER

Most of the time Datasette is used with just a single database file. So maybe it makes sense for this option to be turned on by default and to ALWAYS be available on the Datasette instance homepage unless the user has explicitly disabled it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391584112 https://github.com/simonw/datasette/issues/283#issuecomment-391584112 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTU4NDExMg== simonw 9599 2018-05-24T04:26:29Z 2018-05-24T04:30:50Z OWNER

I built a very rough prototype of this to prove it could work. It's deployed here - and here's an example of a query that joins across two different databases:

https://datasette-cross-database-joins-prototype.now.sh/memory?sql=select+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.rowid%2C%0D%0Afivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.actors%2C%0D%0A%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D.city%0D%0Afrom+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D%0D%0Ajoin+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D%0D%0A++on+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D.rowid+%3D+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.rowid

select fivethirtyeight.[love-actually/love_actually_adjacencies].rowid,
fivethirtyeight.[love-actually/love_actually_adjacencies].actors,
[google-trends].[20150430_UKDebate].city
from fivethirtyeight.[love-actually/love_actually_adjacencies]
join [google-trends].[20150430_UKDebate]
  on [google-trends].[20150430_UKDebate].rowid = fivethirtyeight.[love-actually/love_actually_adjacencies].rowid

I deployed it like this:

datasette publish now --branch=cross-database-joins fivethirtyeight.db google-trends.db --name=datasette-cross-database-joins-prototype
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391584527 https://github.com/simonw/datasette/issues/283#issuecomment-391584527 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTU4NDUyNw== simonw 9599 2018-05-24T04:29:40Z 2018-05-24T04:29:40Z OWNER

Rather than stealing the /memory namespace for this it would be nicer if these cross-database joins could be executed at the very top-level URL of the Datasette instance - https://example.com/?sql=...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391584366 https://github.com/simonw/datasette/issues/283#issuecomment-391584366 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTU4NDM2Ng== simonw 9599 2018-05-24T04:28:20Z 2018-05-24T04:28:20Z OWNER

I used some pretty ugly hacks, like faking an entire .inspect() block for the :memory: database just to get past the errors I was seeing. To ship this as a feature it will need quite a bit of code refactoring to make those hacks unnecessary.

https://github.com/simonw/datasette/blob/7a3040f5782375373b2b66e5969bc2c49b3a6f0e/datasette/views/database.py#L18-L26

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  
391583528 https://github.com/simonw/datasette/issues/283#issuecomment-391583528 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTU4MzUyOA== simonw 9599 2018-05-24T04:21:49Z 2018-05-24T04:21:49Z OWNER

The challenge here is which database should be the "default" database. The first database attached to SQLite is treated as the default - if no database is specified in a query, that's the database that queries will be executed against.

Currently, each database URL in Datasette (e.g. https://san-francisco.datasettes.com/sf-film-locations-84594a7 v.s. https://san-francisco.datasettes.com/sf-trees-ebc2ad9 ) gets its own independent connection, and all queries within that base URL run against that database.

If we're going to attach multiple databases to the same connection, how do we set which database gets to be the default?

The easiest thing to do here will be to have a special database (maybe which is turned off by default and can be enabled using datasette serve --enable-cross-database-joins or similar) which attaches to ALL the databases. Perhaps it starts as an in-memory database, maybe at /memory?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);