issue_comments

Table actions

7 rows where issue = 1426379903 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

author_association 3

issue 1

  • don't use immutable=1, only mode=ro · 7 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1745568725 https://github.com/simonw/datasette/pull/1870#issuecomment-1745568725 https://api.github.com/repos/simonw/datasette/issues/1870 IC_kwDOBm6k_c5oC0PV jdangerx 2495794 2023-10-03T19:12:37Z 2023-10-03T19:12:37Z NONE

Hello! Resurrecting this issue since we're running into something similar with data.catalyst.coop as our database files have ballooned up to several GB. Our Cloud Run revisions now require huge amounts of RAM to start up without receiving a SIGBUS.

I'd love to see this fix merged in. It sounds like we want to make the immutable/read-only mode decision more flexible before doing so, so that we can use ro in Docker and immutable outside. If that sounds right, I'm happy to take a crack at adding that as a command-line flag or something that gets set automatically based on the expected execution environment.

 
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 1
}
 don't use immutable=1, only mode=ro 1426379903  
1295667649 https://github.com/simonw/datasette/pull/1870#issuecomment-1295667649 https://api.github.com/repos/simonw/datasette/issues/1870 IC_kwDOBm6k_c5NOlHB fgregg 536941 2022-10-29T00:52:43Z 2022-10-29T00:53:43Z CONTRIBUTOR

Are you saying that I can build a container, but then when I run it and it does datasette serve -i data.db ... it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that serve command?

Somehow, datasette serve -i data.db will lead to the data.db being modified, which will trigger a copy-on-write of data.db into the read-write layer of the container.

I don't understand how that happens.

it kind of feels like a bug in sqlite, but i can't quite follow the sqlite code.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 don't use immutable=1, only mode=ro 1426379903  
1295660092 https://github.com/simonw/datasette/pull/1870#issuecomment-1295660092 https://api.github.com/repos/simonw/datasette/issues/1870 IC_kwDOBm6k_c5NOjQ8 simonw 9599 2022-10-29T00:25:26Z 2022-10-29T00:25:26Z OWNER

Saw your comment here too: https://github.com/simonw/datasette/issues/1480#issuecomment-1271101072

switching from immutable=1 to mode=ro completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details.

So maybe we need a special case for containers that are intended to be run using Docker - the ones produced by datasette package and datasette publish cloudrun? Those are cases where the -i option should actually be opened in read-only mode, not immutable mode.

Maybe a datasette serve --irw data.db option for opening a file in immutable-but-actually-read-only mode? Bit ugly though.

I should run some benchmarks to figure out if immutable really does offer significant performance benefits.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 don't use immutable=1, only mode=ro 1426379903  
1295657771 https://github.com/simonw/datasette/pull/1870#issuecomment-1295657771 https://api.github.com/repos/simonw/datasette/issues/1870 IC_kwDOBm6k_c5NOisr simonw 9599 2022-10-29T00:19:03Z 2022-10-29T00:19:03Z OWNER

Just saw your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1272357976

when you are running from docker, you always will want to run as mode=ro because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when datasette serve runs.

I don't understand this. My mental model of how Docker works is that the image itself is created using docker build... but then when the image runs later on (docker run) the image itself isn't touched at all.

Are you saying that I can build a container, but then when I run it and it does datasette serve -i data.db ... it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that serve command?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 don't use immutable=1, only mode=ro 1426379903  
1294285471 https://github.com/simonw/datasette/pull/1870#issuecomment-1294285471 https://api.github.com/repos/simonw/datasette/issues/1870 IC_kwDOBm6k_c5NJTqf fgregg 536941 2022-10-28T01:06:03Z 2022-10-28T01:06:03Z CONTRIBUTOR

as far as i can tell, this is where the "immutable" argument is used in sqlite:

c pPager->noLock = sqlite3_uri_boolean(pPager->zFilename, "nolock", 0); if( (iDc & SQLITE_IOCAP_IMMUTABLE)!=0 || sqlite3_uri_boolean(pPager->zFilename, "immutable", 0) ){ vfsFlags |= SQLITE_OPEN_READONLY; goto act_like_temp_file; }

so it does set the read only flag, but then has a goto.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 don't use immutable=1, only mode=ro 1426379903  
1294238862 https://github.com/simonw/datasette/pull/1870#issuecomment-1294238862 https://api.github.com/repos/simonw/datasette/issues/1870 IC_kwDOBm6k_c5NJISO codecov[bot] 22429695 2022-10-27T23:44:25Z 2022-10-27T23:44:25Z NONE

Codecov Report

Base: 92.55% // Head: 92.55% // No change to project coverage :thumbsup:

Coverage data is based on head (4faa4fd) compared to base (bf00b0b). Patch has no changes to coverable lines.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1870 +/- ## ======================================= Coverage 92.55% 92.55% ======================================= Files 35 35 Lines 4432 4432 ======================================= Hits 4102 4102 Misses 330 330 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1870/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.30% <ø> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 don't use immutable=1, only mode=ro 1426379903  
1294237783 https://github.com/simonw/datasette/pull/1870#issuecomment-1294237783 https://api.github.com/repos/simonw/datasette/issues/1870 IC_kwDOBm6k_c5NJIBX fgregg 536941 2022-10-27T23:42:18Z 2022-10-27T23:42:18Z CONTRIBUTOR 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 don't use immutable=1, only mode=ro 1426379903  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);