issue_comments

So there are two sides to supporting this:

• Being able to sensibly introspect composite foreign keys
• Being able to define composite foreign keys when creating a table

Relevant discussion: https://github.com/simonw/sqlite-generate/issues/8#issuecomment-648438056

Here's what's missing: In [11]: db.conn.execute('PRAGMA foreign_key_list(song)').fetchall() Out[11]: [(0, 0, 'album', 'songartist', 'albumartist', 'NO ACTION', 'NO ACTION', 'NONE'), (0, 1, 'album', 'songalbum', 'albumname', 'NO ACTION', 'NO ACTION', 'NONE')] Compare with this code here: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/sqlite_utils/db.py#L563-L579

The first two columns returned by PRAGMA foreign_key_list(table) are id and seq - these show when two foreign key records are part of the same compound foreign key. sqlite-utils entirely ignores those at the moment.

New docs: https://github.com/simonw/sqlite-utils/blob/2.10.1/docs/python-api.rst#introspection

Should go in this section https://sqlite-utils.readthedocs.io/en/stable/python-api.html#introspection - under .columns_dict and before .foreign_keys.

I wonder if this is a SQLite caching issue then?

Datasette has a configuration option for this but I haven't spent much time experimenting with it so I don't know how much of an impact it can have: https://datasette.readthedocs.io/en/stable/config.html#cache-size-kb

I'm chunking inserts at 100 at a time right now: https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L1030

I think the performance is more down to using Faker to create the test data - generating millions of entirely fake, randomized records takes a fair bit of time.

Just to check: are you seeing the problem on this page: https://latest.datasette.io/fixtures (the database page) - or this page (the table page): https://latest.datasette.io/fixtures/compound_three_primary_keys

If it's the table page then the problem may well be #862.

Related: #862 - a time limit on the total time spent considering suggested facets for a table.

Should this be controlled be a separate configuration setting? I'm inclined to say no - I think instead I'll set the limit to be 10 * whatever facet_suggest_time_limit_ms is.

sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50

Looks like that will take 35 minutes to run (it's not a particularly fast tool).

I generated a 600MB database using sqlite-generate just now - with 100 tables at 100,00 rows and 3 tables at 1,000,000 rows - and performance of the database page was fine, 250ms.

Those tables only had 4 columns each though.

You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.

https://github.com/simonw/sqlite-generate is now ready to be used - see also https://pypi.org/project/sqlite-generate/