{"html_url": "https://github.com/simonw/datasette/issues/419#issuecomment-493110184", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/419", "id": 493110184, "node_id": "MDEyOklzc3VlQ29tbWVudDQ5MzExMDE4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-05-16T15:14:31Z", "updated_at": "2019-05-16T15:14:31Z", "author_association": "OWNER", "body": "This is done bar the documentation, which is tracked in #421 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421551434, "label": "Default to opening files in mutable mode, special option for immutable files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/419#issuecomment-473726527", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/419", "id": 473726527, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzcyNjUyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-17T23:28:41Z", "updated_at": "2019-05-16T14:54:50Z", "author_association": "OWNER", "body": "I've added the `-i` option, so this now works:\r\n\r\n    datasette -i fixtures.db\r\n\r\nThis feature is incomplete though. Some extra changes I need to make:\r\n\r\n* The `?_hash=1` and `--config hash_urls:1` options (introduced in #418) should only work for immutable databases #471\r\n* Would be useful if there was a debug screen that could show which databases were mounted as mutable v.s. immutable - maybe a `/-/databases` page? - #470 \r\n* Need to rework how `.inspect()` works, see #420 \r\n* Documentation is needed #421\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421551434, "label": "Default to opening files in mutable mode, special option for immutable files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/419#issuecomment-473708724", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/419", "id": 473708724, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzcwODcyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-17T19:55:21Z", "updated_at": "2019-05-16T03:35:59Z", "author_association": "OWNER", "body": "Thinking about this further: I think I may have made a mistake establishing \"immutable\" as the default mode for databases opened by Datasette.\r\n\r\nWhat would it look like if files were NOT opened in immutable mode by default?\r\n\r\nMaybe the command to start Datasette looks like this:\r\n\r\n    datasette mutable1.db mutable2.db --immutable=this_is_immutable.db --immutable=this_is_immutable2.db\r\n\r\nSo regular file arguments are treated as mutable (and opened in `?mode=ro`) while file arguments passed using the new `--immutable` option are opened in immutable mode.\r\n\r\nThe `-i` shortcut has not yet been taken, so this could be abbreviated to:\r\n\r\n    datasette mutable1.db mutable2.db -i this_is_immutable.db -i this_is_immutable2.db", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421551434, "label": "Default to opening files in mutable mode, special option for immutable files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/419#issuecomment-492903398", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/419", "id": 492903398, "node_id": "MDEyOklzc3VlQ29tbWVudDQ5MjkwMzM5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-05-16T03:33:01Z", "updated_at": "2019-05-16T03:33:01Z", "author_association": "OWNER", "body": "@russss sorry I only just spotted your comment here.\r\n\r\nI think I have an alternative suggestion for what you need to do here. It sounds to me like you need to calculate a specific piece of information against a specific database. Instead of doing this in inspect, how about having a separate tool which runs this once against the database file and writes the result into a database file there?\r\n\r\nI've been thinking about this pattern a bit as part of the sqlite-utils work I've been doing. It's already something that's needed for SQLite FTS support - it's no good just creating a FTS index, you have to populate it as well. In sqlite-utils world you do that like this: https://sqlite-utils.readthedocs.io/en/latest/cli.html#configuring-full-text-search\r\n\r\n    $ sqlite-utils enable-fts mydb.db documents title summary\r\n\r\nBut then later if you've inserted new records you have to call this:\r\n\r\n    $ sqlite-utils populate-fts mydb.db documents title summary\r\n\r\nSo one option here could be for `datasette-geo` to know to look for a special `datasette_geo_bounding_box` database table and, if it's missing, to calculate at runtime (probably once on startup and then cache it).\r\n\r\nAnother option: Datasette now has an option to open a database file in \"immutable\" mode, using `datasette -i mydatabase.db`. When you do that we calculate counts on startup - and we'll also be able to load counts from the `inspect-data.json` file (that's pretty much all that will be in there). I'm open to making this available as a plugin hook - all kinds of optimizations could be run against these `-i` databases. It would essentially be what we have with inspect today but just for databases opened in that specific mode.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421551434, "label": "Default to opening files in mutable mode, special option for immutable files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/419#issuecomment-489060765", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/419", "id": 489060765, "node_id": "MDEyOklzc3VlQ29tbWVudDQ4OTA2MDc2NQ==", "user": {"value": 45057, "label": "russss"}, "created_at": "2019-05-03T11:07:42Z", "updated_at": "2019-05-03T11:07:42Z", "author_association": "CONTRIBUTOR", "body": "Are you planning on removing inspect entirely? \r\n\r\nI didn't spot this work before I started on datasette-geo, but ironically I think it has a use case which really needs the inspect functionality (or some replacement). \r\n\r\nDatasette-geo uses it to store the bounding box of all the geographic features in the table. This is needed when rendering the map because it avoids having to send loads of tile requests for areas which are empty. \r\n\r\nEven with relatively small datasets, calculating the bounding box seems to take around 5 seconds, so I don't think it's really feasible to do this on page load.\r\n\r\nOne possible fix would be to do this on startup, and then in a thread which watches the database for changes.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421551434, "label": "Default to opening files in mutable mode, special option for immutable files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/419#issuecomment-488528111", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/419", "id": 488528111, "node_id": "MDEyOklzc3VlQ29tbWVudDQ4ODUyODExMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-05-02T01:14:58Z", "updated_at": "2019-05-02T01:14:58Z", "author_association": "OWNER", "body": "I just closed #420 - all of the places in the codebase that were using `.inspect()` should have been eliminated.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421551434, "label": "Default to opening files in mutable mode, special option for immutable files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/419#issuecomment-473713363", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/419", "id": 473713363, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzcxMzM2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-17T20:49:39Z", "updated_at": "2019-03-17T20:52:46Z", "author_association": "OWNER", "body": "And a really important difference: the whole model of caching inspect data no longer works for mutable files, because another process might make a change to the database schema (adding a new table for example).\r\n\r\nhttps://fivethirtyeight.datasettes.com/-/inspect\r\n\r\nSo everywhere that uses `self.ds.inspect()` right now will have to change to calling a routine which knows the difference between mutable and immutable databases and queries for live schema data for mutables while using a cache for immutables.\r\n\r\nI'll track this as a separate ticket.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421551434, "label": "Default to opening files in mutable mode, special option for immutable files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/419#issuecomment-473712820", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/419", "id": 473712820, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzcxMjgyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-17T20:43:23Z", "updated_at": "2019-03-17T20:43:51Z", "author_association": "OWNER", "body": "So the differences here are:\r\n\r\n* For immutable databases we calculate content hash and table counts; mutable databases we do not\r\n* Immutable databasse open with `file:{}?immutable=1`, mutable databases open with `file:{}?mode=ro`\r\n* Anywhere that shows a table count now needs to call a new method which knows to run `count(*)` with a timeout for mutable databases, read from the precalculated counts for immutable databases\r\n* The url-hash option should no longer be available at all for mutable databases\r\n* New command-line tool syntax: `datasette mutable.db` v.s. `datasette -i immutable.db`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421551434, "label": "Default to opening files in mutable mode, special option for immutable files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/419#issuecomment-473709883", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/419", "id": 473709883, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzcwOTg4Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-17T20:09:47Z", "updated_at": "2019-03-17T20:37:45Z", "author_association": "OWNER", "body": "Could I persist the last calculated count for a table and somehow detect if that table has been changed in any way by another process, hence invalidating the cached count (and potentially scheduling a new count)?\r\n\r\nhttps://www.sqlite.org/c3ref/update_hook.html says that `sqlite3_update_hook()` can be used to register a handler invoked on almost all update/insert/delete operations to a specific table... except that it misses out on deletes triggered by `ON CONFLICT REPLACE` and only works for `ROWID` tables.\r\n\r\nAlso this hook is not exposed in the Python `sqlite3` library - though it may be available using some terrifying `ctypes` hacks: https://stackoverflow.com/a/16920926\r\n\r\nSo on further research, I think the answer is *no*: I should assume that it won't be possible to cache counts and magically invalidate the cache when the underlying file is changed by another process.\r\n\r\nInstead I need to assume that counts will be an expensive operation.\r\n\r\nAs such, I can introduce a time limit on counts and use that anywhere a count is displayed. If the time limit is exceeded by the `count(*)` query I can show \"many\" instead.\r\n\r\nThat said... running `count(*)` against a table with 200,000 rows in only takes about 3ms, so even a timeout of 20ms is likely to work fine for tables of around a million rows.\r\n\r\nIt would be really neat if I could generate a lower bound count in a limited amount of time. If I counted up to 4m rows before the timeout I could show \"more than 4m rows\". No idea if that would be possible though.\r\n\r\nRelevant: https://stackoverflow.com/questions/8988915/sqlite-count-slow-on-big-tables - reports of very slow counts on 6GB database file. Consensus seems to be \"yeah, that's just how SQLite is built\" - though there was a suggestion that you can use `select max(ROWID) from table` provided you are certain there have been no deletions.\r\n\r\nAlso relevant: http://sqlite.1065341.n5.nabble.com/sqlite3-performance-on-select-count-very-slow-for-16-GB-file-td80176.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421551434, "label": "Default to opening files in mutable mode, special option for immutable files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/419#issuecomment-473708941", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/419", "id": 473708941, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzcwODk0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-17T19:58:11Z", "updated_at": "2019-03-17T19:58:11Z", "author_association": "OWNER", "body": "Some problems to solve:\r\n\r\n* Right now Datasette assumes it can always show the count of rows in a table, because this has been pre-calculated. If a database is mutable the pre-calculation trick no longer works, and for giant tables a `select count(*) from X` query can be expensive to run. Maybe we set a time limit on these? If time limit expires show \"many rows\"?\r\n* Maintaining a content hash of the table no longer makes sense if it is changing (though interestingly there's a `.sha3sum` built-in SQLite CLI command which takes a hash of the content and stays the same even through vacuum runs). Without that we need a different mechanism for calculating table colours. It also means that we can't do the special dbname-hash URL trick (see #418) at all if the database is opened as mutable.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421551434, "label": "Default to opening files in mutable mode, special option for immutable files"}, "performed_via_github_app": null}