Demo is now live here: https://latest.datasette.io/_memory

The documentation is at https://docs.datasette.io/en/latest/sql_queries.html#cross-database-queries - it links to this example query: https://latest.datasette.io/_memory?sql=select%0D%0A++%27fixtures%27+as+database%2C+%0D%0Afrom%0D%0A++%5Bfixtures%5D.sqlite_master%0D%0Aunion%0D%0Aselect%0D%0A++%27extra_database%27+as+database%2C+%0D%0Afrom%0D%0A++%5Bextra_database%5D.sqlite_master

Codecov Report

Merging #1232 (8876499) into main (4df548e) will increase coverage by 0.03%. The diff coverage is 100.00%.

```diff @@ Coverage Diff @@

main #1232 +/-

========================================== + Coverage 91.42% 91.46% +0.03%
========================================== Files 32 32
Lines 3955 3970 +15
========================================== + Hits 3616 3631 +15
Misses 339 339
```

| Impacted Files | Coverage Δ | | |---|---|---| | datasette/app.py | 95.68% <100.00%> (+0.06%) | :arrow_up: | | datasette/cli.py | 76.62% <100.00%> (+0.36%) | :arrow_up: | | datasette/views/database.py | 97.19% <100.00%> (+0.01%) | :arrow_up: |

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 4df548e...8876499. Read the comment docs.

The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run datasette one.db two.db three.db --crossdb and then use the /_memory page to run joins across tables from multiple databases.

It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see Datasette Library) this won't be particularly useful for you.

So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting /_memory?attach=db1&attach=db2 to get a special connection.

Also worth noting: plugins that implement the prepare_connection() hook can attach additional databases - so if you need better, customized support for this one way to handle that would be with a custom plugin.

UI listing the attached tables:

I tested the demo deployment like this: datasette publish cloudrun fixtures.db extra_database.db \ -m fixtures.json \ --plugins-dir=plugins \ --branch=crossdb \ --extra-options="--setting template_debug 1 --crossdb" \ --install=pysqlite3-binary \ --service=datasette-latest-crossdb

Due to bug #1233 I'm going to publish the additional database as extra_database.db rather than extra database.db as it is used in the tests.

I think the bug is here: https://github.com/simonw/datasette/blob/640ac7071b73111ba4423812cd683756e0e1936b/datasette/utils/init.py#L349-L373

I'll document this option on https://docs.datasette.io/en/stable/sql_queries.html

The new warning looks like this:

It would also be neat if https://latest.datasette.io/ had multiple databases attached in order to demonstrate this feature.

It would be neat if the /_memory page showed a list of attached databases, to indicate that the --crossdb option is working and give people links to click to start running queries.

I have a working version now, moving development to a pull request.

For the moment I'm going to hard-code a SQLITE_LIMIT_ATTACHED=10 constant and only attach the first 10 databases to the _memory connection.

select * from pragma_database_list(); is useful - shows all attached databases for the current connection.

It turns out SQLite defaults to a maximum of 10 attached databases. This can be increased using a compile-time constant, but even with that it cannot be more than 62: https://stackoverflow.com/questions/9845448/attach-limit-10

Ideally I'd figure out a way to replicate this error in a concurrent unit test.

I started trying to use locks to resolve this but I've not figured out the right way to do that yet - here's my first experiment: ```diff diff --git a/datasette/app.py b/datasette/app.py index 9e15a16..1681c9d 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -217,6 +217,7 @@ class Datasette: self.inspect_data = inspect_data self.immutables = set(immutables or []) self.databases = collections.OrderedDict() + self._refresh_schemas_lock = threading.Lock() if memory or not self.files: self.add_database(Database(self, is_memory=True), name="_memory") # memory_name is a random string so that each Datasette instance gets its own @@ -324,6 +325,13 @@ class Datasette: self.client = DatasetteClient(self)

 async def refresh_schemas(self):

• return
• if self._refresh_schemas_lock.locked():
• return
• with self._refresh_schemas_lock:
• await self._refresh_schemas() +
• async def _refresh_schemas(self): internal_db = self.databases["_internal"] if not self.internal_db_created: await init_internal_db(internal_db) ```

This broke CI: https://github.com/simonw/datasette/runs/1929355965?check_suite_focus=true

I can use click.IntRange(min=None, max=None) for this. https://click.palletsprojects.com/en/7.x/options/#ranges - inclusive on both edges.

Awesome!

[Edit] Oh, I just saw the "Load all" button under the cluster map as well as the setting to alter the max number or results. So I guess this issue only is about the Vega charts.

I'm going to to try prototyping the --crossdb option that causes /_memory to connect to all databases as a starting point and see how well that works.

I was going ask you about this issue when we talk during your office-hours schedule this Friday, but was there any support ever added for doing this cross-database joining?

I have a use-case where could be pretty neat to do analysis using this tool on time-specific databases from snapshots

https://ilsweb.cincinnatilibrary.org/collection-analysis/

and thanks again for such an amazing tool!