Updated documentation: https://docs.datasette.io/en/latest/settings.html#configuration-directory-mode

... and here's @ocdtrekkie's plugin! https://github.com/ocdtrekkie/datasette-external-links-new-tabs

@simonw, should i open up a new issue for investigating the differences between "immutable=1" and "mode=ro" and possibly switching to "mode=ro". Or would you like to keep that conversation in this issue?

switching from immutable=1 to mode=ro completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details.

yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in immutable mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!

running a test of that now.

this completely addressed #1480

Codecov Report

Base: 92.50% // Head: 92.51% // Increases project coverage by +0.01% :tada:

Coverage data is based on head (9bead2a) compared to base (eff1124). Patch coverage: 100.00% of modified lines in pull request are covered.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

the only mode that publish cloudrun supports right now is immutable

when i hack the connect method to open non mutable files with "mode=ro" and not "immutable=1" https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L79

then:

bash 870 B RUN /bin/sh -c datasette inspect nlrb.db --inspect-file inspect-data.json

the datasette inspect layer is only the size of the json file!

Ooh, I didn't even think about links in tables! You're definitely right on the approach to this. It might also be a really good "stupidly simple" plugin for me to try to build myself, which could be fun.

The argument that has always convinced me NOT to use target="_blank" (even for links like this one) is that it breaks browser expectations.

If you click a link with `target="_blank" on it you get a new browser window... with a disabled back button. You have to then know to close that browser window in order to return to the previous page - as opposed to hitting the "back" button like usual.

You'll note that Datasette doesn't use target="_blank" even on URLs presented in database tables - like these ones: https://latest.datasette.io/fixtures/roadside_attractions

So I'm very firmly in the anti-target-blank camp!

This is the kind of change which I'd suggest implementing as a plugin. datasette-external-links-new-windows could run a bit of JavaScript on every page that looks for <a href="..."> elements that link to off-domain pages and adds target="_blank" to them via the DOM.

That way people who like target="_blank" can have it!

yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in immutable mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!

running a test of that now.

Just overlapped with your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1271003212 - which notes that opening with ?immutable=1 DOES seem to cause the file to be duplicated!

Oh this is interesting!

Is your hunch here that running this line is causing the file to be stored as a second layer?

https://github.com/simonw/datasette/blob/5aa359b86907d11b3ee601510775a85a90224da8/datasette/utils/init.py#L399

I guess it's possible that running a non-read-only query against the database causes one or two bytes to be changed (maybe a transaction ID or similar?)

Modifying the inspect command to use ?mode=ro seems sensible to me. Except.... it should already be opening those files in immutable mode according to this line: https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/cli.py#L172-L173

Here's what opening as a immutables does:

https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/app.py#L258-L260

https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L98

and if we try immutable mode, which is how things are opened by datasette inspect we duplicate the files!!!

```python

test_sql_immutable.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?immutable=1', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

fascinatingly, telling python to open sqlite in read only mode makes this layer have a size of 0

```python

test_sql_ro.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?mode=ro', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

that's quite weird because setting the file permissions to read only didn't do anything. (on reflection, that chmod isn't doing anything because the dockerfile commands are run as root)

okay, some progress!! running some sql against a database file causes that file to get duplicated even if it doesn't apparently change the file.

make a little test script like this:

```python

test_sql.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

then

docker RUN python test_sql.py nlrb.db

produced a layer that's the same size as nlrb.db!!

it's not that the inspect command is somehow changing the db files. if i set them to only read-only, the "inspect" layer still has the same very large size.

i thought it was maybe to do with reading through all the files, but that does not seem to be the case

if i make a little test file like:

```python

test_read.py

import hashlib import sys import pathlib

HASH_BLOCK_SIZE = 1024 * 1024

def inspect_hash(path): """Calculate the hash of a database, efficiently.""" m = hashlib.sha256() with path.open("rb") as fp: while True: data = fp.read(HASH_BLOCK_SIZE) if not data: break m.update(data)

return m.hexdigest()

inspect_hash(pathlib.Path(sys.argv[1])) ```

then a line in the Dockerfile like

docker RUN python test_read.py nlrb.db && echo "[]" > /etc/inspect.json

just produes a layer of 3B

Codecov Report

Base: 92.50% // Head: 92.50% // No change to project coverage :thumbsup:

Coverage data is based on head (c12447e) compared to base (eff1124). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.