home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

27 rows where issue = 1079149656 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • simonw 27

issue 1

  • Optimize all those calls to index_list and foreign_key_list · 27 ✖

author_association 1

  • OWNER 27
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
997459958 https://github.com/simonw/datasette/issues/1555#issuecomment-997459958 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47dAf2 simonw 9599 2021-12-19T20:55:59Z 2021-12-19T20:55:59Z OWNER

Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997325189 https://github.com/simonw/datasette/issues/1555#issuecomment-997325189 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cfmF simonw 9599 2021-12-19T03:55:01Z 2021-12-19T20:54:51Z OWNER

It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in .execute_fn(). I wonder if there's something smart I can do about that - maybe have trace() record that function with a traceback even though it doesn't have the executed SQL string?

5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997459637 https://github.com/simonw/datasette/issues/1555#issuecomment-997459637 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47dAa1 simonw 9599 2021-12-19T20:53:46Z 2021-12-19T20:53:46Z OWNER

Using #1571 showed me that the DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? and table_name = ? queries were running way more times than I expected. I came up with a new optimization that just does DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? instead.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997342494 https://github.com/simonw/datasette/issues/1555#issuecomment-997342494 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cj0e simonw 9599 2021-12-19T07:22:04Z 2021-12-19T07:22:04Z OWNER

Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997324666 https://github.com/simonw/datasette/issues/1555#issuecomment-997324666 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cfd6 simonw 9599 2021-12-19T03:47:51Z 2021-12-19T03:48:09Z OWNER

Here's a hacked together prototype of running all of that stuff inside a single function passed to .execute_fn():

```diff diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 95055d8..58f9982 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -1,4 +1,5 @@ import textwrap +from datasette.utils import table_column_details

async def init_internal_db(db): @@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db): "DELETE FROM tables WHERE database_name = ?", [database_name], block=True ) tables = (await db.execute("select * from sqlite_master WHERE type = 'table'")).rows - tables_to_insert = [] - columns_to_delete = [] - columns_to_insert = [] - foreign_keys_to_delete = [] - foreign_keys_to_insert = [] - indexes_to_delete = [] - indexes_to_insert = []

  • for table in tables:
  • table_name = table["name"]
  • tables_to_insert.append(
  • (database_name, table_name, table["rootpage"], table["sql"])
  • )
  • columns_to_delete.append((database_name, table_name))
  • columns = await db.table_column_details(table_name)
  • columns_to_insert.extend(
  • {
  • **{"database_name": database_name, "table_name": table_name},
  • **column._asdict(),
  • }
  • for column in columns
  • )
  • foreign_keys_to_delete.append((database_name, table_name))
  • foreign_keys = (
  • await db.execute(f"PRAGMA foreign_key_list([{table_name}])")
  • ).rows
  • foreign_keys_to_insert.extend(
  • {
  • **{"database_name": database_name, "table_name": table_name},
  • **dict(foreign_key),
  • }
  • for foreign_key in foreign_keys
  • )
  • indexes_to_delete.append((database_name, table_name))
  • indexes = (await db.execute(f"PRAGMA index_list([{table_name}])")).rows
  • indexes_to_insert.extend(
  • {
  • **{"database_name": database_name, "table_name": table_name},
  • **dict(index),
  • }
  • for index in indexes
  • def collect_info(conn):
  • tables_to_insert = []
  • columns_to_delete = []
  • columns_to_insert = []
  • foreign_keys_to_delete = []
  • foreign_keys_to_insert = []
  • indexes_to_delete = []
  • indexes_to_insert = [] +
  • for table in tables:
  • table_name = table["name"]
  • tables_to_insert.append(
  • (database_name, table_name, table["rootpage"], table["sql"])
  • )
  • columns_to_delete.append((database_name, table_name))
  • columns = table_column_details(conn, table_name)
  • columns_to_insert.extend(
  • {
  • **{"database_name": database_name, "table_name": table_name},
  • **column._asdict(),
  • }
  • for column in columns
  • )
  • foreign_keys_to_delete.append((database_name, table_name))
  • foreign_keys = conn.execute(
  • f"PRAGMA foreign_key_list([{table_name}])"
  • ).fetchall()
  • foreign_keys_to_insert.extend(
  • {
  • **{"database_name": database_name, "table_name": table_name},
  • **dict(foreign_key),
  • }
  • for foreign_key in foreign_keys
  • )
  • indexes_to_delete.append((database_name, table_name))
  • indexes = conn.execute(f"PRAGMA index_list([{table_name}])").fetchall()
  • indexes_to_insert.extend(
  • {
  • **{"database_name": database_name, "table_name": table_name},
  • **dict(index),
  • }
  • for index in indexes
  • )
  • return (
  • tables_to_insert,
  • columns_to_delete,
  • columns_to_insert,
  • foreign_keys_to_delete,
  • foreign_keys_to_insert,
  • indexes_to_delete,
  • indexes_to_insert, )

  • (

  • tables_to_insert,
  • columns_to_delete,
  • columns_to_insert,
  • foreign_keys_to_delete,
  • foreign_keys_to_insert,
  • indexes_to_delete,
  • indexes_to_insert,
  • ) = await db.execute_fn(collect_info) + await internal_db.execute_write_many( """ INSERT INTO tables (database_name, table_name, rootpage, sql) ``` First impressions: it looks like this helps a lot - as far as I can tell this is now taking around 21ms to get to the point at which all of those internal databases have been populated, where previously it took more than 180ms.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997324156 https://github.com/simonw/datasette/issues/1555#issuecomment-997324156 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cfV8 simonw 9599 2021-12-19T03:40:05Z 2021-12-19T03:40:05Z OWNER

Using the prototype of this: - https://github.com/simonw/datasette-pretty-traces/issues/5

I'm seeing about 180ms spent running all of these queries on startup!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997321767 https://github.com/simonw/datasette/issues/1555#issuecomment-997321767 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cewn simonw 9599 2021-12-19T03:10:58Z 2021-12-19T03:10:58Z OWNER

I wonder how much overhead there is switching between the async event loop main code and the thread that runs the SQL queries.

Would there be a performance boost if I gathered all of the column/index information in a single function run on the thread using db.execute_fn() I wonder? It would eliminate a bunch of switching between threads.

Would be great to understand how much of an impact that would have.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997321653 https://github.com/simonw/datasette/issues/1555#issuecomment-997321653 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47ceu1 simonw 9599 2021-12-19T03:09:43Z 2021-12-19T03:09:43Z OWNER

On that same documentation page I just spotted this:

This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported.

This makes me nervous to rely on pragma function optimizations in Datasette itself.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997321477 https://github.com/simonw/datasette/issues/1555#issuecomment-997321477 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cesF simonw 9599 2021-12-19T03:07:33Z 2021-12-19T03:07:33Z OWNER

If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old PRAGMA foreign_key_list(table) stuff working for those older versions.

That's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997321327 https://github.com/simonw/datasette/issues/1555#issuecomment-997321327 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cepv simonw 9599 2021-12-19T03:05:39Z 2021-12-19T03:05:44Z OWNER

This caught me out once before in: - https://github.com/simonw/datasette/issues/1276

Turns out Glitch was running SQLite 3.11.0 from 2016-02-15.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997321217 https://github.com/simonw/datasette/issues/1555#issuecomment-997321217 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47ceoB simonw 9599 2021-12-19T03:04:16Z 2021-12-19T03:04:16Z OWNER

One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc

The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997321115 https://github.com/simonw/datasette/issues/1555#issuecomment-997321115 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cemb simonw 9599 2021-12-19T03:03:12Z 2021-12-19T03:03:12Z OWNER

Table columns is a bit harder, because table_xinfo is only in SQLite 3.26.0 or higher: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/utils/init.py#L565-L581

So if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

sql SELECT sqlite_master.name, table_xinfo.* FROM sqlite_master, pragma_table_xinfo(sqlite_master.name) AS table_xinfo WHERE sqlite_master.type = 'table' And otherwise, using table_info: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_info.*%2C%0D%0A++0+as+hidden%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_info%28sqlite_master.name%29+AS+table_info%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

sql SELECT sqlite_master.name, table_info.*, 0 as hidden FROM sqlite_master, pragma_table_info(sqlite_master.name) AS table_info WHERE sqlite_master.type = 'table'

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997320824 https://github.com/simonw/datasette/issues/1555#issuecomment-997320824 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47ceh4 simonw 9599 2021-12-19T02:59:57Z 2021-12-19T03:00:44Z OWNER

To list all indexes: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++index_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_index_list%28sqlite_master.name%29+AS+index_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

sql SELECT sqlite_master.name, index_list.* FROM sqlite_master, pragma_index_list(sqlite_master.name) AS index_list WHERE sqlite_master.type = 'table'

Foreign keys: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++foreign_key_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_foreign_key_list%28sqlite_master.name%29+AS+foreign_key_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27

sql SELECT sqlite_master.name, foreign_key_list.* FROM sqlite_master, pragma_foreign_key_list(sqlite_master.name) AS foreign_key_list WHERE sqlite_master.type = 'table'

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997272223 https://github.com/simonw/datasette/issues/1555#issuecomment-997272223 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cSqf simonw 9599 2021-12-18T19:17:13Z 2021-12-18T19:17:13Z OWNER

That's a good optimization. Still need to deal with the huge flurry of PRAGMA queries though before I can consider this done.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997267416 https://github.com/simonw/datasette/issues/1555#issuecomment-997267416 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cRfY simonw 9599 2021-12-18T18:44:53Z 2021-12-18T18:45:28Z OWNER

Rather than adding a executemany=True parameter, I'm now thinking a better design might be to have three methods:

  • db.execute_write(sql, params=None, block=False)
  • db.execute_writescript(sql, block=False)
  • db.execute_writemany(sql, params_seq, block=False)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997266100 https://github.com/simonw/datasette/issues/1555#issuecomment-997266100 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cRK0 simonw 9599 2021-12-18T18:40:02Z 2021-12-18T18:40:02Z OWNER

The implementation of cursor.executemany() looks very efficient - it turns into a call to this C function with multiple set to 1: https://github.com/python/cpython/blob/e002bbc6cce637171fb2b1391ffeca8643a13843/Modules/_sqlite/cursor.c#L468-L469

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997262475 https://github.com/simonw/datasette/issues/1555#issuecomment-997262475 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cQSL simonw 9599 2021-12-18T18:34:18Z 2021-12-18T18:34:18Z OWNER

Using executescript=True that call now takes 1.89ms to create all of those tables.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997248364 https://github.com/simonw/datasette/issues/1555#issuecomment-997248364 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cM1s simonw 9599 2021-12-18T18:20:10Z 2021-12-18T18:20:10Z OWNER

Idea: teach execute_write to accept an optional executescript=True parameter, like this: ```diff diff --git a/datasette/database.py b/datasette/database.py index 468e936..1a424f5 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -94,10 +94,14 @@ class Database: f"file:{self.path}{qs}", uri=True, check_same_thread=False )

  • async def execute_write(self, sql, params=None, block=False):
  • async def execute_write(self, sql, params=None, executescript=False, block=False):
  • assert not executescript and params, "Cannot use params with executescript=True" def _inner(conn): with conn:
  • return conn.execute(sql, params or [])
  • if executescript:
  • return conn.executescript(sql)
  • else:
  • return conn.execute(sql, params or [])
     with trace("sql", database=self.name, sql=sql.strip(), params=params):
         results = await self.execute_write_fn(_inner, block=block)
    

    ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997245301 https://github.com/simonw/datasette/issues/1555#issuecomment-997245301 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cMF1 simonw 9599 2021-12-18T18:17:04Z 2021-12-18T18:17:04Z OWNER

One downside of conn.executescript() is that it won't be picked up by the tracing mechanism - in fact nothing that uses await db.execute_write_fn(fn, block=True) or await db.execute_fn(fn, block=True) gets picked up by tracing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997241969 https://github.com/simonw/datasette/issues/1555#issuecomment-997241969 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cLRx simonw 9599 2021-12-18T18:13:04Z 2021-12-18T18:13:04Z OWNER

Also: running all of those CREATE TABLE IF NOT EXISTS in a single call to conn.executescript() rather than as separate queries may speed things up too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997241645 https://github.com/simonw/datasette/issues/1555#issuecomment-997241645 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cLMt simonw 9599 2021-12-18T18:12:26Z 2021-12-18T18:12:26Z OWNER

A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling .execute_write() in a loop.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997235086 https://github.com/simonw/datasette/issues/1555#issuecomment-997235086 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cJmO simonw 9599 2021-12-18T17:30:13Z 2021-12-18T17:30:13Z OWNER

Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997234858 https://github.com/simonw/datasette/issues/1555#issuecomment-997234858 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47cJiq simonw 9599 2021-12-18T17:28:44Z 2021-12-18T17:28:44Z OWNER

Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster.

I'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data.

So maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997128508 https://github.com/simonw/datasette/issues/1555#issuecomment-997128508 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47bvk8 simonw 9599 2021-12-18T02:33:57Z 2021-12-18T02:33:57Z OWNER

Here's why - trace only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997128368 https://github.com/simonw/datasette/issues/1555#issuecomment-997128368 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47bviw simonw 9599 2021-12-18T02:32:43Z 2021-12-18T02:32:43Z OWNER

I wonder why the INSERT INTO queries don't show up in that ?trace=1 view?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997128251 https://github.com/simonw/datasette/issues/1555#issuecomment-997128251 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47bvg7 simonw 9599 2021-12-18T02:31:51Z 2021-12-18T02:31:51Z OWNER

I was thinking it might even be possible to convert this into a insert into tables select from ... query:

https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L102-L112

But the SELECT runs against a separate database from the INSERT INTO, so I would have to setup a cross-database connection for this which feels a little too complicated.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  
997128080 https://github.com/simonw/datasette/issues/1555#issuecomment-997128080 https://api.github.com/repos/simonw/datasette/issues/1555 IC_kwDOBm6k_c47bveQ simonw 9599 2021-12-18T02:30:19Z 2021-12-18T02:30:19Z OWNER

I think all of these queries happen in one place - in the populate_schema_tables() function - so optimizing them might be localized to just that area of the code, which would be nice:

https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L97-L183

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Optimize all those calls to index_list and foreign_key_list 1079149656  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Queries took 1.2ms · About: github-to-sqlite