{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-905900807", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 905900807, "node_id": "IC_kwDOBm6k_c41_vMH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T21:51:10Z", "updated_at": "2021-08-25T21:51:10Z", "author_association": "OWNER", "body": "10-20 minutes to populate `_internal`! How many databases and tables is that for?\r\n\r\nI may have to rethink the `_internal` mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed.\r\n\r\nI need to get my own stress testing rig setup for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-648234787", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 648234787, "node_id": "MDEyOklzc3VlQ29tbWVudDY0ODIzNDc4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-23T15:22:51Z", "updated_at": "2020-06-23T15:22:51Z", "author_association": "OWNER", "body": "I wonder if this is a SQLite caching issue then?\n\nDatasette has a configuration option for this but I haven't spent much time experimenting with it so I don't know how much of an impact it can have: https://datasette.readthedocs.io/en/stable/config.html#cache-size-kb", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-648163272", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 648163272, "node_id": "MDEyOklzc3VlQ29tbWVudDY0ODE2MzI3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-23T13:52:23Z", "updated_at": "2020-06-23T13:52:23Z", "author_association": "OWNER", "body": "I'm chunking inserts at 100 at a time right now: https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L1030\r\n\r\nI think the performance is more down to using Faker to create the test data - generating millions of entirely fake, randomized records takes a fair bit of time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-647894903", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 647894903, "node_id": "MDEyOklzc3VlQ29tbWVudDY0Nzg5NDkwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-23T04:07:59Z", "updated_at": "2020-06-23T04:07:59Z", "author_association": "OWNER", "body": "Just to check: are you seeing the problem on this page: https://latest.datasette.io/fixtures (the database page) - or this page (the table page): https://latest.datasette.io/fixtures/compound_three_primary_keys\r\n\r\nIf it's the table page then the problem may well be #862.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-647890619", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 647890619, "node_id": "MDEyOklzc3VlQ29tbWVudDY0Nzg5MDYxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-23T03:48:21Z", "updated_at": "2020-06-23T03:48:21Z", "author_association": "OWNER", "body": "    sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50\r\n\r\nLooks like that will take 35 minutes to run (it's not a particularly fast tool).\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-647890378", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 647890378, "node_id": "MDEyOklzc3VlQ29tbWVudDY0Nzg5MDM3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-23T03:47:19Z", "updated_at": "2020-06-23T03:47:19Z", "author_association": "OWNER", "body": "I generated a 600MB database using [sqlite-generate](https://github.com/simonw/sqlite-generate) just now - with 100 tables at 100,00 rows and 3 tables at 1,000,000 rows - and performance of the database page was fine, 250ms.\r\n\r\nThose tables only had 4 columns each though.\r\n\r\nYou said \"200k+, 50+ rows in a couple of tables\" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-647189948", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 647189948, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NzE4OTk0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-21T22:30:12Z", "updated_at": "2020-06-21T22:30:43Z", "author_association": "OWNER", "body": "I'll write a little script which generates a 300MB SQLite file with a bunch of tables with lots of randomly generated rows in to help test this.\r\n\r\nHaving a tool like that which can generate larger databases with different gnarly performance characteristics will be useful for other performance work too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-647189666", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 647189666, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NzE4OTY2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-21T22:26:55Z", "updated_at": "2020-06-21T22:26:55Z", "author_association": "OWNER", "body": "This makes a lot of sense. I implemented the mechanism for the index page because I have my own instance of Datasette that was running slow, but it had a dozen database files attached to it. I've not run into this with a single giant database file but it absolutely makes sense that the same optimization would be necessary for the database page there too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}