issue_comments

So there are two sides to supporting this:

• Being able to sensibly introspect composite foreign keys
• Being able to define composite foreign keys when creating a table

Relevant discussion: https://github.com/simonw/sqlite-generate/issues/8#issuecomment-648438056

Here's what's missing: In [11]: db.conn.execute('PRAGMA foreign_key_list(song)').fetchall() Out[11]: [(0, 0, 'album', 'songartist', 'albumartist', 'NO ACTION', 'NO ACTION', 'NONE'), (0, 1, 'album', 'songalbum', 'albumname', 'NO ACTION', 'NO ACTION', 'NONE')] Compare with this code here: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/sqlite_utils/db.py#L563-L579

The first two columns returned by PRAGMA foreign_key_list(table) are id and seq - these show when two foreign key records are part of the same compound foreign key. sqlite-utils entirely ignores those at the moment.

New docs: https://github.com/simonw/sqlite-utils/blob/2.10.1/docs/python-api.rst#introspection

Should go in this section https://sqlite-utils.readthedocs.io/en/stable/python-api.html#introspection - under .columns_dict and before .foreign_keys.

@simonw

Did you find the reason? I had similar situation and I check this on millions ways. I am sure app doesn't consume such memory.

I was trying the app with: docker run --rm -it -p 80:80 -m 128M foo

I was watching app with docker stats. Even limited memory by CMD ["java", "-Xms60M", "-Xmx60M", "-jar", "api.jar"]. Checked memory usage by app in code and print bash commands. The app definitely doesn't use this memory. Also doesn't write files.

Only one solution is to change memory to 512M.

It is definitely something wrong with cloud run.

I even did special app for testing this. It looks like when I cross very small amount of code / memory / app size in random when, then memory needs grow +hundreds. Nothing make sense here. Especially it works everywhere expect cloud run.

Please let me know if you discovered something more.

I wonder if this is a SQLite caching issue then?

Datasette has a configuration option for this but I haven't spent much time experimenting with it so I don't know how much of an impact it can have: https://datasette.readthedocs.io/en/stable/config.html#cache-size-kb

The issue seems to appear sporadically, like when I return to database page after a while, during which some records have been added to the database.

I've just visited database, page first visit took ~10s, consecutive visits took 0.3s.

I'm chunking inserts at 100 at a time right now: https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L1030

I think the performance is more down to using Faker to create the test data - generating millions of entirely fake, randomized records takes a fair bit of time.

Hmm, not seeing the problem now.
I've removed the commented out sections in database.py and restarted the process. Database page now loads in <250ms.

I have couple of workers that check some pages regularly and scrape new content and save to the DB. Could it be that datasette tries to recount tables every time database size changes? Normally it keeps a count cache, but as DB gets updated so often (new content every 5 min or so) it's practically recounting every time I go to the database page?

EDIT: It turns out it doesn't hold cache with mutable databases.

I'll update the issue with more findings and a better way to reproduce the problem if I encounter it again.

sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50

Looks like that will take 35 minutes to run (it's not a particularly fast tool).

Try chunking write operations into batches every 1000 records or so.

You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.

Ah that was a typo, I meant 50k.

I think I should mention that having FTS on all tables mean I have 5 visible, 25 hidden (FTS) tables displayed on database page.

Just to check: are you seeing the problem on this page: https://latest.datasette.io/fixtures (the database page) - or this page (the table page): https://latest.datasette.io/fixtures/compound_three_primary_keys

If it's the table page then the problem may well be #862.

Related: #862 - a time limit on the total time spent considering suggested facets for a table.

Should this be controlled be a separate configuration setting? I'm inclined to say no - I think instead I'll set the limit to be 10 * whatever facet_suggest_time_limit_ms is.

sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50

Looks like that will take 35 minutes to run (it's not a particularly fast tool).

I generated a 600MB database using sqlite-generate just now - with 100 tables at 100,00 rows and 3 tables at 1,000,000 rows - and performance of the database page was fine, 250ms.

Those tables only had 4 columns each though.

You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.

https://github.com/simonw/sqlite-generate is now ready to be used - see also https://pypi.org/project/sqlite-generate/