html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/859#issuecomment-905900807,https://api.github.com/repos/simonw/datasette/issues/859,905900807,IC_kwDOBm6k_c41_vMH,9599,simonw,2021-08-25T21:51:10Z,2021-08-25T21:51:10Z,OWNER,"10-20 minutes to populate `_internal`! How many databases and tables is that for?

I may have to rethink the `_internal` mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed.

I need to get my own stress testing rig setup for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-648234787,https://api.github.com/repos/simonw/datasette/issues/859,648234787,MDEyOklzc3VlQ29tbWVudDY0ODIzNDc4Nw==,9599,simonw,2020-06-23T15:22:51Z,2020-06-23T15:22:51Z,OWNER,"I wonder if this is a SQLite caching issue then?

Datasette has a configuration option for this but I haven't spent much time experimenting with it so I don't know how much of an impact it can have: https://datasette.readthedocs.io/en/stable/config.html#cache-size-kb","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-648163272,https://api.github.com/repos/simonw/datasette/issues/859,648163272,MDEyOklzc3VlQ29tbWVudDY0ODE2MzI3Mg==,9599,simonw,2020-06-23T13:52:23Z,2020-06-23T13:52:23Z,OWNER,"I'm chunking inserts at 100 at a time right now: https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L1030

I think the performance is more down to using Faker to create the test data - generating millions of entirely fake, randomized records takes a fair bit of time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647894903,https://api.github.com/repos/simonw/datasette/issues/859,647894903,MDEyOklzc3VlQ29tbWVudDY0Nzg5NDkwMw==,9599,simonw,2020-06-23T04:07:59Z,2020-06-23T04:07:59Z,OWNER,"Just to check: are you seeing the problem on this page: https://latest.datasette.io/fixtures (the database page) - or this page (the table page): https://latest.datasette.io/fixtures/compound_three_primary_keys

If it's the table page then the problem may well be #862.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647890619,https://api.github.com/repos/simonw/datasette/issues/859,647890619,MDEyOklzc3VlQ29tbWVudDY0Nzg5MDYxOQ==,9599,simonw,2020-06-23T03:48:21Z,2020-06-23T03:48:21Z,OWNER,"    sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50

Looks like that will take 35 minutes to run (it's not a particularly fast tool).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647890378,https://api.github.com/repos/simonw/datasette/issues/859,647890378,MDEyOklzc3VlQ29tbWVudDY0Nzg5MDM3OA==,9599,simonw,2020-06-23T03:47:19Z,2020-06-23T03:47:19Z,OWNER,"I generated a 600MB database using [sqlite-generate](https://github.com/simonw/sqlite-generate) just now - with 100 tables at 100,00 rows and 3 tables at 1,000,000 rows - and performance of the database page was fine, 250ms.

Those tables only had 4 columns each though.

You said ""200k+, 50+ rows in a couple of tables"" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647189948,https://api.github.com/repos/simonw/datasette/issues/859,647189948,MDEyOklzc3VlQ29tbWVudDY0NzE4OTk0OA==,9599,simonw,2020-06-21T22:30:12Z,2020-06-21T22:30:43Z,OWNER,"I'll write a little script which generates a 300MB SQLite file with a bunch of tables with lots of randomly generated rows in to help test this.

Having a tool like that which can generate larger databases with different gnarly performance characteristics will be useful for other performance work too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647189666,https://api.github.com/repos/simonw/datasette/issues/859,647189666,MDEyOklzc3VlQ29tbWVudDY0NzE4OTY2Ng==,9599,simonw,2020-06-21T22:26:55Z,2020-06-21T22:26:55Z,OWNER,"This makes a lot of sense. I implemented the mechanism for the index page because I have my own instance of Datasette that was running slow, but it had a dozen database files attached to it. I've not run into this with a single giant database file but it absolutely makes sense that the same optimization would be necessary for the database page there too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),