github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/issues/1646#issuecomment-1272149176 | https://api.github.com/repos/simonw/datasette/issues/1646 | 1272149176 | IC_kwDOBm6k_c5L03S4 | 9599 | 2022-10-07T23:06:17Z | 2022-10-07T23:06:17Z | OWNER | Updated documentation: https://docs.datasette.io/en/latest/settings.html#configuration-directory-mode | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1157182254 | |
https://github.com/simonw/datasette/pull/1838#issuecomment-1271803298 | https://api.github.com/repos/simonw/datasette/issues/1838 | 1271803298 | IC_kwDOBm6k_c5Lzi2i | 9599 | 2022-10-07T16:28:41Z | 2022-10-07T16:28:41Z | OWNER | ... and here's @ocdtrekkie's plugin! https://github.com/ocdtrekkie/datasette-external-links-new-tabs | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400494162 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1271103097 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1271103097 | IC_kwDOBm6k_c5Lw355 | 536941 | 2022-10-07T04:43:41Z | 2022-10-07T04:43:41Z | CONTRIBUTOR | @simonw, should i open up a new issue for investigating the differences between "immutable=1" and "mode=ro" and possibly switching to "mode=ro". Or would you like to keep that conversation in this issue? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/issues/1480#issuecomment-1271101072 | https://api.github.com/repos/simonw/datasette/issues/1480 | 1271101072 | IC_kwDOBm6k_c5Lw3aQ | 536941 | 2022-10-07T04:39:10Z | 2022-10-07T04:39:10Z | CONTRIBUTOR | switching from `immutable=1` to `mode=ro` completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1015646369 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1271100651 | IC_kwDOBm6k_c5Lw3Tr | 536941 | 2022-10-07T04:38:14Z | 2022-10-07T04:38:14Z | CONTRIBUTOR | > yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in `immutable` mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation! > > running a test of that now. this completely addressed #1480 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/pull/1820#issuecomment-1258601033 | https://api.github.com/repos/simonw/datasette/issues/1820 | 1258601033 | IC_kwDOBm6k_c5LBLpJ | 22429695 | 2022-09-26T20:32:47Z | 2022-10-07T03:58:13Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.50**% // Head: **92.51**% // Increases project coverage by **`+0.01%`** :tada: > Coverage data is based on head [(`9bead2a`)](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`eff1124`)](https://codecov.io/gh/simonw/datasette/commit/eff112498ecc499323c26612d707908831446d25?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 100.00% of modified lines in pull request are covered. <details><summary>Additional details and impacted files</summary> ```diff @@ Coverage Diff @@ ## main #1820 +/- ## ========================================== + Coverage 92.50% 92.51% +0.01% ========================================== Files 35 35 Lines 4400 4406 +6 ========================================== + Hits 4070 4076 +6 Misses 330 330 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.11% <ø> (ø)` | | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `94.80% <10… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1386456717 | |
https://github.com/simonw/datasette/issues/1301#issuecomment-1271035998 | https://api.github.com/repos/simonw/datasette/issues/1301 | 1271035998 | IC_kwDOBm6k_c5Lwnhe | 536941 | 2022-10-07T02:38:04Z | 2022-10-07T02:38:04Z | CONTRIBUTOR | the only mode that `publish cloudrun` supports right now is immutable | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
860722711 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1271020193 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1271020193 | IC_kwDOBm6k_c5Lwjqh | 536941 | 2022-10-07T02:15:05Z | 2022-10-07T02:21:08Z | CONTRIBUTOR | when i hack the connect method to open non mutable files with "mode=ro" and not "immutable=1" https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L79 then: ```bash 870 B RUN /bin/sh -c datasette inspect nlrb.db --inspect-file inspect-data.json ``` the `datasette inspect` layer is only the size of the json file! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/pull/1838#issuecomment-1271024708 | https://api.github.com/repos/simonw/datasette/issues/1838 | 1271024708 | IC_kwDOBm6k_c5LwkxE | 4399499 | 2022-10-07T02:19:49Z | 2022-10-07T02:19:49Z | NONE | Ooh, I didn't even think about links in tables! You're definitely right on the approach to this. It might also be a really good "stupidly simple" plugin for me to try to build myself, which could be fun. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400494162 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1271008997 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1271008997 | IC_kwDOBm6k_c5Lwg7l | 536941 | 2022-10-07T02:00:37Z | 2022-10-07T02:00:49Z | CONTRIBUTOR | yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in `immutable` mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation! running a test of that now. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1271006020 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1271006020 | IC_kwDOBm6k_c5LwgNE | 9599 | 2022-10-07T01:54:07Z | 2022-10-07T01:54:07Z | OWNER | Just overlapped with your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1271003212 - which notes that opening with `?immutable=1` DOES seem to cause the file to be duplicated! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1271004167 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1271004167 | IC_kwDOBm6k_c5LwfwH | 9599 | 2022-10-07T01:53:05Z | 2022-10-07T01:53:05Z | OWNER | Oh this is interesting! Is your hunch here that running this line is causing the file to be stored as a second layer? https://github.com/simonw/datasette/blob/5aa359b86907d11b3ee601510775a85a90224da8/datasette/utils/__init__.py#L399 I guess it's possible that running a non-read-only query against the database causes one or two bytes to be changed (maybe a transaction ID or similar?) Modifying the `inspect` command to use `?mode=ro` seems sensible to me. Except.... it should already be opening those files in immutable mode according to this line: https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/cli.py#L172-L173 Here's what opening as a `immutables` does: https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/app.py#L258-L260 https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L98 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1271003212 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1271003212 | IC_kwDOBm6k_c5LwfhM | 536941 | 2022-10-07T01:52:04Z | 2022-10-07T01:52:04Z | CONTRIBUTOR | and if we try immutable mode, which is how things are opened by `datasette inspect` we duplicate the files!!! ```python # test_sql_immutable.py import sqlite3 import sys db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?immutable=1', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1270992795 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1270992795 | IC_kwDOBm6k_c5Lwc-b | 536941 | 2022-10-07T01:29:15Z | 2022-10-07T01:50:14Z | CONTRIBUTOR | fascinatingly, telling python to open sqlite in read only mode makes this layer have a size of 0 ```python # test_sql_ro.py import sqlite3 import sys db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?mode=ro', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ``` that's quite weird because setting the file permissions to read only didn't do anything. (on reflection, that chmod isn't doing anything because the dockerfile commands are run as root) | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1270988081 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1270988081 | IC_kwDOBm6k_c5Lwb0x | 536941 | 2022-10-07T01:19:01Z | 2022-10-07T01:27:35Z | CONTRIBUTOR | okay, some progress!! running some sql against a database file causes that file to get duplicated even if it doesn't apparently change the file. make a little test script like this: ```python # test_sql.py import sqlite3 import sys db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ``` then ```docker RUN python test_sql.py nlrb.db ``` produced a layer that's the same size as `nlrb.db`!! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1270936982 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1270936982 | IC_kwDOBm6k_c5LwPWW | 536941 | 2022-10-07T00:52:41Z | 2022-10-07T00:52:41Z | CONTRIBUTOR | it's not that the inspect command is somehow changing the db files. if i set them to only read-only, the "inspect" layer still has the same very large size. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/issues/1836#issuecomment-1270923537 | https://api.github.com/repos/simonw/datasette/issues/1836 | 1270923537 | IC_kwDOBm6k_c5LwMER | 536941 | 2022-10-07T00:46:08Z | 2022-10-07T00:46:08Z | CONTRIBUTOR | i thought it was maybe to do with reading through all the files, but that does not seem to be the case if i make a little test file like: ```python # test_read.py import hashlib import sys import pathlib HASH_BLOCK_SIZE = 1024 * 1024 def inspect_hash(path): """Calculate the hash of a database, efficiently.""" m = hashlib.sha256() with path.open("rb") as fp: while True: data = fp.read(HASH_BLOCK_SIZE) if not data: break m.update(data) return m.hexdigest() inspect_hash(pathlib.Path(sys.argv[1])) ``` then a line in the Dockerfile like ```docker RUN python test_read.py nlrb.db && echo "[]" > /etc/inspect.json ``` just produes a layer of `3B` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400374908 | |
https://github.com/simonw/datasette/pull/1837#issuecomment-1270855853 | https://api.github.com/repos/simonw/datasette/issues/1837 | 1270855853 | IC_kwDOBm6k_c5Lv7it | 22429695 | 2022-10-07T00:01:20Z | 2022-10-07T00:01:20Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1837?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.50**% // Head: **92.50**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`c12447e`)](https://codecov.io/gh/simonw/datasette/pull/1837?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`eff1124`)](https://codecov.io/gh/simonw/datasette/commit/eff112498ecc499323c26612d707908831446d25?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines. <details><summary>Additional details and impacted files</summary> ```diff @@ Coverage Diff @@ ## main #1837 +/- ## ======================================= Coverage 92.50% 92.50% ======================================= Files 35 35 Lines 4400 4400 ======================================= Hits 4070 4070 Misses 330 330 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) </details> [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1837?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1400431789 |