issue_comments

It feels slightly weird continuing to call it metadata.json as it starts to grow support for config options (which already started with the units and facets keys) but I can live with that.

So the lookup priority order should be:

• table level in metadata
• database level in metadata
• root level in metadata
• --config options passed to datasette serve
• DATASETTE_X environment variables

I like /-/all-5de27e3 for this (with /-/all redirecting to the correct hash)

Maybe allow_sql, allow_facet and allow_download

This will also give us a mechanism for turning on and off the cross-database joins feature from #283

I'm not crazy about the enable_ prefix on these.

For an example query that pre-populates that textarea... maybe a UNION that pulls the first 10 rows from the first table of each of the first two databases?

select * from (select rowid, actors from fivethirtyeight.[love-actually/love_actually_adjacencies] limit 10) union all select * from (select rowid, city from [google-trends].[20150430_UKDebate] limit 10)

https://datasette-cross-database-joins-prototype.now.sh/memory?sql=select++from+%28select+rowid%2C+actors+from+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D+limit+10%29%0D%0A+++union+all%0D%0Aselect++from+%28select+rowid%2C+city+from+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D+limit+10%29

On the /-all-5de27e3 page we can show the regular https://fivethirtyeight.datasettes.com/fivethirtyeight-5de27e3 interface but instead of the list of tables we can show a list of attached databases plus some help text showing how to construct a cross-database join.

Giving it /all/ would be easier since that way the existing URL routes (including canned queries) would all work... but I would have to teach it NOT to expect a database content hash on that URL.

Or maybe it should still have a content hash (to enable far-future cache expiry headers on query results) but the hash should be constructed out of all of the other database hashes concatenated together.

That way the URLs would be /all-5de27e3 and /all-5de27e3/canned-query-name

Only downside: this would make it impossible to have a database file with the name all.db. I think that's probably an OK trade-off. You could turn the feature off with a config flag if you really want to use that filename (for whatever reason).

How about /-all-5de27e3/ instead to avoid collisions?

Another option: give this the /-/all URL namespace.

Should this support canned queries too? I think it should, though that raises interesting questions regarding their URL structure.

This would make Datasett's SQL features a lot more instantly obvious to people who land on a homepage, which is probably a good thing.

Most of the time Datasette is used with just a single database file. So maybe it makes sense for this option to be turned on by default and to ALWAYS be available on the Datasette instance homepage unless the user has explicitly disabled it.

I built a very rough prototype of this to prove it could work. It's deployed here - and here's an example of a query that joins across two different databases:

https://datasette-cross-database-joins-prototype.now.sh/memory?sql=select+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.rowid%2C%0D%0Afivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.actors%2C%0D%0A%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D.city%0D%0Afrom+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D%0D%0Ajoin+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D%0D%0A++on+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D.rowid+%3D+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.rowid

select fivethirtyeight.[love-actually/love_actually_adjacencies].rowid, fivethirtyeight.[love-actually/love_actually_adjacencies].actors, [google-trends].[20150430_UKDebate].city from fivethirtyeight.[love-actually/love_actually_adjacencies] join [google-trends].[20150430_UKDebate] on [google-trends].[20150430_UKDebate].rowid = fivethirtyeight.[love-actually/love_actually_adjacencies].rowid I deployed it like this:

datasette publish now --branch=cross-database-joins fivethirtyeight.db google-trends.db --name=datasette-cross-database-joins-prototype

Rather than stealing the /memory namespace for this it would be nicer if these cross-database joins could be executed at the very top-level URL of the Datasette instance - https://example.com/?sql=...

I used some pretty ugly hacks, like faking an entire .inspect() block for the :memory: database just to get past the errors I was seeing. To ship this as a feature it will need quite a bit of code refactoring to make those hacks unnecessary.

https://github.com/simonw/datasette/blob/7a3040f5782375373b2b66e5969bc2c49b3a6f0e/datasette/views/database.py#L18-L26

The challenge here is which database should be the "default" database. The first database attached to SQLite is treated as the default - if no database is specified in a query, that's the database that queries will be executed against.

Currently, each database URL in Datasette (e.g. https://san-francisco.datasettes.com/sf-film-locations-84594a7 v.s. https://san-francisco.datasettes.com/sf-trees-ebc2ad9 ) gets its own independent connection, and all queries within that base URL run against that database.

If we're going to attach multiple databases to the same connection, how do we set which database gets to be the default?

The easiest thing to do here will be to have a special database (maybe which is turned off by default and can be enabled using datasette serve --enable-cross-database-joins or similar) which attaches to ALL the databases. Perhaps it starts as an in-memory database, maybe at /memory?