Added some useful new documented internal methods in: - #1570

That's a good optimization. Still need to deal with the huge flurry of PRAGMA queries though before I can consider this done.

This will replace the work done in #1569.

Rather than adding a executemany=True parameter, I'm now thinking a better design might be to have three methods:

• db.execute_write(sql, params=None, block=False)
• db.execute_writescript(sql, block=False)
• db.execute_writemany(sql, params_seq, block=False)

Updated documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-executescript-false-block-false

The implementation of cursor.executemany() looks very efficient - it turns into a call to this C function with multiple set to 1: https://github.com/python/cpython/blob/e002bbc6cce637171fb2b1391ffeca8643a13843/Modules/_sqlite/cursor.c#L468-L469

Using executescript=True that call now takes 1.89ms to create all of those tables.

Goal here is to gain the ability to use conn.executescript() and still have it show up in the tracer.

https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executescript

Idea: teach execute_write to accept an optional executescript=True parameter, like this: ```diff diff --git a/datasette/database.py b/datasette/database.py index 468e936..1a424f5 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -94,10 +94,14 @@ class Database: f"file:{self.path}{qs}", uri=True, check_same_thread=False )

• async def execute_write(self, sql, params=None, block=False):
• async def execute_write(self, sql, params=None, executescript=False, block=False):
• assert not executescript and params, "Cannot use params with executescript=True" def _inner(conn): with conn:
• return conn.execute(sql, params or [])
• if executescript:
• return conn.executescript(sql)
• else:
• return conn.execute(sql, params or [])

   with trace("sql", database=self.name, sql=sql.strip(), params=params):
       results = await self.execute_write_fn(_inner, block=block)
  

  ```

One downside of conn.executescript() is that it won't be picked up by the tracing mechanism - in fact nothing that uses await db.execute_write_fn(fn, block=True) or await db.execute_fn(fn, block=True) gets picked up by tracing.

Also: running all of those CREATE TABLE IF NOT EXISTS in a single call to conn.executescript() rather than as separate queries may speed things up too.

A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling .execute_write() in a loop.

I can release a new version of datasette-leaflet-freedraw as soon as this is out.

Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized:

Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster.

I'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data.

So maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance!

Now running at https://latest-with-plugins.datasette.io/github/commits?_trace=1

Prototype:

```diff diff --git a/datasette/database.py b/datasette/database.py index 0a0c104..468e936 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -99,7 +99,9 @@ class Database: with conn: return conn.execute(sql, params or [])

• return await self.execute_write_fn(_inner, block=block)
• with trace("sql", database=self.name, sql=sql.strip(), params=params):
• results = await self.execute_write_fn(_inner, block=block)

• return results

  async def execute_write_fn(self, fn, block=False): task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io") ```

Here's why - trace only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211

I wonder why the INSERT INTO queries don't show up in that ?trace=1 view?

I was thinking it might even be possible to convert this into a insert into tables select from ... query:

https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L102-L112

But the SELECT runs against a separate database from the INSERT INTO, so I would have to setup a cross-database connection for this which feels a little too complicated.

I think all of these queries happen in one place - in the populate_schema_tables() function - so optimizing them might be localized to just that area of the code, which would be nice:

https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L97-L183

Oh that's an interesting solution, combining the hashes of all of the individual databases.

I'm actually not a big fan of hashed_url mode - I implemented it right at the start of the project because it felt like a clever hack, and then ended up making it not-the-default a few years ago:

• 418

• 419

• 421

I've since not found myself wanting to use it at all for any of my projects - which makes me nervous, because it means there's a pretty complex feature that I'm not using at all, so it's only really protected by the existing unit tests for it.

What I'd really like to do is figure out how to have hashed URL mode work entirely as a plugin - then I could extract it from Datasette core entirely (which would simplify a bunch of stuff) but people who find the optimization useful would be able to access it.

I'm not sure that the existing plugin hooks are robust enough to do that yet though.

Docs here: https://docs.datasette.io/en/latest/internals.html#datasette-class

I should document the usage of this constructor in https://docs.datasette.io/en/stable/internals.html#datasette-class

Sure - there are actually several levels to this.

The code that creates connections to the database is this: https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/database.py#L72-L95

For files on disk, it does this: ```python

For read-only connections

conn = sqlite3.connect( "file:my.db?mode=ro", uri=True, check_same_thread=False)

For connections that should be treated as immutable:

conn = sqlite3.connect( "file:my.db?immutable=1", uri=True, check_same_thread=False) For in-memory databases it runs this after the connection has been created:python conn.execute("PRAGMA query_only=1") `` SQLitePRAGMAqueries are treated as dangerous: someone could runPRAGMA query_only=0` to turn that previous option off for example.

So this function runs against any incoming SQL to verify that it looks like a SELECT ... and doesn't have anything like that in it.

https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/utils/init.py#L195-L204

You can see the tests for that here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/tests/test_utils.py#L136-L170

Made this change while working on this issue: - #1567

I'm going to write a test for this that uses that sleep() SQL function from c35b84a2aabe2f14aeacf6cda4110ae1e94d6059.

I want to get this into Datasette 0.60 - #1566 - it's a small change that can unlock a lot of potential.

I refactored this code out into the filters.py module in aa7f0037a46eb76ae6fe9bf2a1f616c58738ecdf

As noted in https://github.com/simonw/datasette/issues/621#issuecomment-552253208 a common pattern in this method is blocks of code that append new items to the where_clauses, params and extra_human_descriptions arrays. This is a useful refactoring opportunity.

Code that fits this pattern:

• The code that builds based on the filters: where_clauses, params = filters.build_where_clauses(table) and human_description_en = filters.human_description_en(extra=extra_human_descriptions)
• Code that handles ?_where=: where_clauses.extend(request.args["_where"]) - though note that this also appends to a extra_wheres_for_ui array which nothing else uses
• The _through= code, see #621 for details
• The code that deals with ?_search= FTS

The keyset pagination code modifies where_clauses and params too, but I don't think it's quite going to work with the same abstraction that would cover the above examples.

[UPDATE December 2021 - this comment became the basis for a new filters_from_request plugin hook, see also #473]