issue_comments

Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment.

It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in .execute_fn(). I wonder if there's something smart I can do about that - maybe have trace() record that function with a traceback even though it doesn't have the executed SQL string?

5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem.

Using #1571 showed me that the DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? and table_name = ? queries were running way more times than I expected. I came up with a new optimization that just does DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? instead.

Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly.

Here's a hacked together prototype of running all of that stuff inside a single function passed to .execute_fn():

```diff diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 95055d8..58f9982 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -1,4 +1,5 @@ import textwrap +from datasette.utils import table_column_details

async def init_internal_db(db): @@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db): "DELETE FROM tables WHERE database_name = ?", [database_name], block=True ) tables = (await db.execute("select * from sqlite_master WHERE type = 'table'")).rows - tables_to_insert = [] - columns_to_delete = [] - columns_to_insert = [] - foreign_keys_to_delete = [] - foreign_keys_to_insert = [] - indexes_to_delete = [] - indexes_to_insert = []

• for table in tables:
• table_name = table["name"]
• tables_to_insert.append(
• (database_name, table_name, table["rootpage"], table["sql"])
• )
• columns_to_delete.append((database_name, table_name))
• columns = await db.table_column_details(table_name)
• columns_to_insert.extend(
• {
• **{"database_name": database_name, "table_name": table_name},
• **column._asdict(),
• }
• for column in columns
• )
• foreign_keys_to_delete.append((database_name, table_name))
• foreign_keys = (
• await db.execute(f"PRAGMA foreign_key_list([{table_name}])")
• ).rows
• foreign_keys_to_insert.extend(
• {
• **{"database_name": database_name, "table_name": table_name},
• **dict(foreign_key),
• }
• for foreign_key in foreign_keys
• )
• indexes_to_delete.append((database_name, table_name))
• indexes = (await db.execute(f"PRAGMA index_list([{table_name}])")).rows
• indexes_to_insert.extend(
• {
• **{"database_name": database_name, "table_name": table_name},
• **dict(index),
• }
• for index in indexes
• def collect_info(conn):
• tables_to_insert = []
• columns_to_delete = []
• columns_to_insert = []
• foreign_keys_to_delete = []
• foreign_keys_to_insert = []
• indexes_to_delete = []
• indexes_to_insert = [] +
• for table in tables:
• table_name = table["name"]
• tables_to_insert.append(
• (database_name, table_name, table["rootpage"], table["sql"])
• )
• columns_to_delete.append((database_name, table_name))
• columns = table_column_details(conn, table_name)
• columns_to_insert.extend(
• {
• **{"database_name": database_name, "table_name": table_name},
• **column._asdict(),
• }
• for column in columns
• )
• foreign_keys_to_delete.append((database_name, table_name))
• foreign_keys = conn.execute(
• f"PRAGMA foreign_key_list([{table_name}])"
• ).fetchall()
• foreign_keys_to_insert.extend(
• {
• **{"database_name": database_name, "table_name": table_name},
• **dict(foreign_key),
• }
• for foreign_key in foreign_keys
• )
• indexes_to_delete.append((database_name, table_name))
• indexes = conn.execute(f"PRAGMA index_list([{table_name}])").fetchall()
• indexes_to_insert.extend(
• {
• **{"database_name": database_name, "table_name": table_name},
• **dict(index),
• }
• for index in indexes
• )
• return (
• tables_to_insert,
• columns_to_delete,
• columns_to_insert,
• foreign_keys_to_delete,
• foreign_keys_to_insert,
• indexes_to_delete,

• indexes_to_insert, )

• (

• tables_to_insert,
• columns_to_delete,
• columns_to_insert,
• foreign_keys_to_delete,
• foreign_keys_to_insert,
• indexes_to_delete,
• indexes_to_insert,
• ) = await db.execute_fn(collect_info) + await internal_db.execute_write_many( """ INSERT INTO tables (database_name, table_name, rootpage, sql) ``` First impressions: it looks like this helps a lot - as far as I can tell this is now taking around 21ms to get to the point at which all of those internal databases have been populated, where previously it took more than 180ms.

Using the prototype of this: - https://github.com/simonw/datasette-pretty-traces/issues/5

I'm seeing about 180ms spent running all of these queries on startup!

I wonder how much overhead there is switching between the async event loop main code and the thread that runs the SQL queries.

Would there be a performance boost if I gathered all of the column/index information in a single function run on the thread using db.execute_fn() I wonder? It would eliminate a bunch of switching between threads.

Would be great to understand how much of an impact that would have.

On that same documentation page I just spotted this:

This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported.

This makes me nervous to rely on pragma function optimizations in Datasette itself.

If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old PRAGMA foreign_key_list(table) stuff working for those older versions.

That's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0.

This caught me out once before in: - https://github.com/simonw/datasette/issues/1276

Turns out Glitch was running SQLite 3.11.0 from 2016-02-15.

One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc

The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature.

Table columns is a bit harder, because table_xinfo is only in SQLite 3.26.0 or higher: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/utils/init.py#L565-L581

So if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

sql SELECT sqlite_master.name, table_xinfo.* FROM sqlite_master, pragma_table_xinfo(sqlite_master.name) AS table_xinfo WHERE sqlite_master.type = 'table' And otherwise, using table_info: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_info.*%2C%0D%0A++0+as+hidden%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_info%28sqlite_master.name%29+AS+table_info%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

sql SELECT sqlite_master.name, table_info.*, 0 as hidden FROM sqlite_master, pragma_table_info(sqlite_master.name) AS table_info WHERE sqlite_master.type = 'table'

To list all indexes: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++index_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_index_list%28sqlite_master.name%29+AS+index_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

sql SELECT sqlite_master.name, index_list.* FROM sqlite_master, pragma_index_list(sqlite_master.name) AS index_list WHERE sqlite_master.type = 'table'

Foreign keys: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++foreign_key_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_foreign_key_list%28sqlite_master.name%29+AS+foreign_key_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

sql SELECT sqlite_master.name, foreign_key_list.* FROM sqlite_master, pragma_foreign_key_list(sqlite_master.name) AS foreign_key_list WHERE sqlite_master.type = 'table'