issue_comments

Efficient row counts are even more important for the DatabaseView and IndexView pages.

The row counts on those pages don't have to be precise, so one option is for me to calculate them and cache them occasionally. I could even have a dedicated thread which just does the counting?

In #422 I've figured out a mechanism for getting accurate or lower-bound counts within a time limit (accurate if possible, lower-bound otherwise).

Still need to solve: TableView.data() - but this is the one with a row count in hence the need to solve #422

expand_foreign_keys() relies on the .inspect() command having automatically derived the label_column for a table, which it does using this code:

https://github.com/simonw/datasette/blob/97331f3435ba1583a0f9dbcaffc25de8894cf1f8/datasette/inspect.py#L34-L42

This needs access to the column names for the table. I think we can drop this entirely in favour of a new utility function - and that function can incorporate the metadata check as well.

This means the Datasette class needs a new property, keeping track of all of the connected databases.

ds.databases = { "name_used_in_urls": { "type": "file", # or "memory" "path": filepath # or None if memory "mutable": True # or False, "hash": "..." # or None if mutable } }

Maybe these should be objects, not dictionaries.

Next I need to fix this:

https://github.com/simonw/datasette/blob/0209a0a344503157351e625f0629b686961763c9/datasette/app.py#L420-L435

Given the name of the database (from the URL e.g. https://latest.datasette.io/fixtures) I need to figure out what name I used to cache the collection.

Even more tricky: table_exists() is currently a synchronous function. If it's going to be executing a SQL query it needs to become an async function.

I started looking at how I would implement table_exists() with a direct call that uses sqlite-utils to see if a table exists.

https://github.com/simonw/datasette/blob/82fec6048148b58748040a7e2caa163387e982a3/datasette/app.py#L303-L304

sqlite-utils needs access to the database connection - but the database connection itself is currently only available in code that runs in a thread inside the .execute() method:

https://github.com/simonw/datasette/blob/82fec6048148b58748040a7e2caa163387e982a3/datasette/app.py#L413-L426

So I'm going to need to refactor this a bit. I think I need a way to say "here is a function which needs access to the connection object for database named X - run that function in a thread, give it access to that connection and then give me back the result".

A microbenchmark against fivethirtyeight.db (415 tables):

In [1]: import sqlite3                                                                                              
In [2]: c = sqlite3.connect("fivethirtyeight.db")                                                                   
In [3]: %timeit c.execute("select name from sqlite_master where type = 'table'").fetchall()                         
283 µs ± 12.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [4]: tables = [r[0] for r in c.execute("select name from sqlite_master where type = 'table'").fetchall()]        
In [5]: len(tables)                                                                                                 
Out[5]: 415
In [6]: %timeit [c.execute("pragma foreign_keys([{}])".format(t)).fetchall() for t in tables]                       
1.81 ms ± 161 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

So running pragma foreign_keys() against 415 tables only takes 1.81ms. This is going to be fine.

Most of these can be replaced with relatively straight-forward direct introspection of the SQLite table.

The one exception is the incoming foreign keys: these can only be found by inspecting ALL of the other tables.

This requires running PRAGMA foreign_key_list([table_name]) against every other table in the database. How expensive is doing this on a database with hundreds of tables?

I systematically reviewed the codebase for things that .inspect() is used for:

In app.py:

• table_exists() uses table in self.inspect().get(database, {}).get("tables")
• .execute() looks up the database name to get the info["file"] (the correct filename with the .db extension)

In cli.py:

• The datasette inspect command dumps it to JSON
• datasette skeleton iterates over it
• datasette serve calls it on startup (to populate static cache of inspect data)

In base.py:

• .database_url(database) calls it to lookup the hash (if hash_urls config turned on)
• .resolve_db_name() uses it to lookup the hash

In database.py:

• DatabaseView uses it to find up the list of tables and views to display, plus the size of the DB file in bytes
• DatabaseDownload uses it to get the filepath for download

In index.py:

• IndexView uses it extensively - to loop through every database and every table. This would make a good starting point for the refactor.

In table.py:

• sortable_columns_for_table() uses it to find the columns in a table
• expandable_columns() uses it to find foreign keys
• expand_foreign_keys() uses it to find foreign keys
• display_columns_and_rows() uses it to find primary keys and foreign keys... but also has access to a cursor.description which it uses to list the columns
• TableView.data uses it to lookup columns and primary keys and the table_rows_count (used if the thing isn't a view) and probably a few more things, this method is huge!
• RowView.data uses it for primary keys
• foreign_key_tables() uses it for foreign keys

In the tests it's used by test_api.test_inspect_json() and by a couple of tests in test_inspect.

Maybe this is a good opportunity to improve the introspection capabilities in sqlite-utils and add it as a dependency.

Needed for #419

Some examples:

https://github.com/simonw/datasette/blob/1f54e092306b208125f39d06712b02895eb75168/datasette/views/table.py#L34-L40

https://github.com/simonw/datasette/blob/1f54e092306b208125f39d06712b02895eb75168/datasette/views/table.py#L45-L48

https://github.com/simonw/datasette/blob/1f54e092306b208125f39d06712b02895eb75168/datasette/views/table.py#L62-L65

https://github.com/simonw/datasette/blob/1f54e092306b208125f39d06712b02895eb75168/datasette/views/table.py#L112-L123

https://github.com/simonw/datasette/blob/1f54e092306b208125f39d06712b02895eb75168/datasette/views/index.py#L11-L19

https://github.com/simonw/datasette/blob/afe9aa3ae03c485c5d6652741438d09445a486c1/datasette/views/base.py#L143-L147