issue_comments

For a demo, visit https://latest.datasette.io/login-as-root and then hit https://latest.datasette.io/_schemas

Needs documentation, but I can wait to write that until I've tested out the feature a bit more.

Getting all the tests to pass is tricky because this adds a whole extra database to Datasette - and there's various code that loops through ds.databases as part of the tests.

I'm going to move the code into a utils/schemas.py module, to avoid further extending the Datasette class definition and to make it more easily testable.

I'm going to tidy this up and land it. A couple of additional decisions:

• The database will be called /_schemas
• By default it will only be visible to root - thus avoiding having to solve the permissions problem with regards to users seeing schemas for tables that are otherwise invisible to them.

I'm not going to block this issue on permissions - I will tackle the efficient bulk permissions problem in #1152.

The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75

A paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.

I may be overthinking that problem. Many queries are fast in SQLite. If a Datasette instance has 1,000 connected tables will even that be a performance problem for permission checks? I should benchmark to find out.

I do need to solve the permissions problem properly though, because one of the goals of this system is to provide a paginated, searchable list of databases and tables for the homepage of the instance - #991.

As such, the homepage will need to be able to display only the tables and databases that the user has permission to view.

One solution on permissions: if Datasette had an efficient way of saying "list the tables that this user has access to" I could use that as a filter any time the user views the schema information. The implementation could be tricky though.

I need to figure out how this will interact with Datasette permissions.

If some tables are private, but others are public, should users be able to see the private tables listed in the schema metadata?

If not, how can that mechanism work?

I'm going to use five tables to start off with:

• databases - a list of databases. Each one has a name, path (if it's on disk), is_memory, schema_version
• tables - a list of tables. Each row is database_name, table_name, sql (the create table statement) - may add more tables in the future, in particular maybe a last_row_count to cache results of counting the rows.
• columns - a list of columns. It's the output of pragma_table_xinfo with the database_name and table_name columns added at the beginning.
• foreign_keys - a list of foreign keys - pragma_foreign_key_list output plus database_name and table_name.
• indexes - a list of indexes - pragma_table_xinfo output plus database_name and table_name.

I could have another table that stores the combined rows from sqlite_máster on every connected database so I have a copy of the schema SQL.

Simpler implementation idea: a Datasette method .refresh_schemas() which loops through all known databases, checks their schema version and updates the in-memory schemas database if they have changed.

I think I'm going to have to build this without using the pragma_x() SQL functions as they were only added in 3.16 in 2017-01-02 and I've seen plenty of Datasette instances running on older versions of SQLite.

Maintaining this database will be the responsibility of a subclass of Database called _SchemaDatabase which will be managed by the Datasette instance.

SQLite uses indexes rather than indices as the plural, so I'll go with that: https://sqlite.org/lang_createindex.html

I've been rediscovering the pattern I already documented in this TIL: https://github.com/simonw/til/blob/main/sqlite/list-all-columns-in-a-database.md#better-alternative-using-a-join

https://www.sqlite.org/pragma.html#pragfunc says:

• This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported.
• The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature.

Here's a simpler query pattern (not using CTEs so should work on older versions of SQLite) - this one lists all indexes for all tables: sql select sqlite_master.name as 'table', indexes.* from sqlite_master join pragma_index_list(sqlite_master.name) indexes where sqlite_master.type = 'table' https://latest.datasette.io/fixtures?sql=select%0D%0A++sqlite_master.name+as+%27table%27%2C%0D%0A++indexes.*%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_index_list%28sqlite_master.name%29+indexes%0D%0Awhere%0D%0A++sqlite_master.type+%3D+%27table%27

This query uses a join to pull foreign key information for every table: https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%0D%0Aselect%0D%0A++tables.name+as+%27table%27%2C%0D%0A++foo.*%0D%0Afrom%0D%0A++tables%0D%0A++join+pragma_foreign_key_list%28tables.name%29+foo

sql with tables as ( select name from sqlite_master where type = 'table' ) select tables.name as 'table', foo.* from tables join pragma_foreign_key_list(tables.name) foo

Same query for pragma_table_xinfo: https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%0D%0Aselect%0D%0A++tables.name+as+%27table%27%2C%0D%0A++foo.*%0D%0Afrom%0D%0A++tables%0D%0A++join+pragma_table_xinfo%28tables.name%29+foo

Prototype: https://latest.datasette.io/fixtures?sql=select+%27facetable%27+as+%27table%27%2C++from+pragma_table_xinfo%28%27facetable%27%29%0D%0Aunion%0D%0Aselect+%27searchable%27+as+%27table%27%2C++from+pragma_table_xinfo%28%27searchable%27%29%0D%0Aunion%0D%0Aselect+%27compound_three_primary_keys%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27compound_three_primary_keys%27%29

sql select 'facetable' as 'table', * from pragma_table_xinfo('facetable') union select 'searchable' as 'table', * from pragma_table_xinfo('searchable') union select 'compound_three_primary_keys' as 'table', * from pragma_table_xinfo('compound_three_primary_keys')

table | cid | name | type | notnull | dflt_value | pk | hidden -- | -- | -- | -- | -- | -- | -- | -- compound_three_primary_keys | 0 | pk1 | varchar(30) | 0 |   | 1 | 0 compound_three_primary_keys | 1 | pk2 | varchar(30) | 0 |   | 2 | 0 compound_three_primary_keys | 2 | pk3 | varchar(30) | 0 |   | 3 | 0 compound_three_primary_keys | 3 | content | text | 0 |   | 0 | 0 facetable | 0 | pk | integer | 0 |   | 1 | 0 facetable | 1 | created | text | 0 |   | 0 | 0 facetable | 2 | planet_int | integer | 0 |   | 0 | 0 facetable | 3 | on_earth | integer | 0 |   | 0 | 0 facetable | 4 | state | text | 0 |   | 0 | 0 facetable | 5 | city_id | integer | 0 |   | 0 | 0 facetable | 6 | neighborhood | text | 0 |   | 0 | 0 facetable | 7 | tags | text | 0 |   | 0 | 0 facetable | 8 | complex_array | text | 0 |   | 0 | 0 facetable | 9 | distinct_some_null |   | 0 |   | 0 | 0 searchable | 0 | pk | integer | 0 |   | 1 | 0 searchable | 1 | text1 | text | 0 |   | 0 | 0 searchable | 2 | text2 | text | 0 |   | 0 | 0 searchable | 3 | name with . and spaces | text | 0 |   | 0 | 0

Next step: design a schema for the in-memory database table that exposes all of the tables.

I want to support things like:

• Show me all of the tables
• Show me the columns in a table
• Show me all tables that contain a tags column
• Show me the indexes
• Show me every table configured for full-text search

Maybe a starting point would be to build concrete tables using the results of things like PRAGMA foreign_key_list(table) and PRAGMA table_xinfo(table) - note though that table_xinfo is SQLite 3.26.0 or higher, as shown here: https://github.com/simonw/datasette/blob/5e9895c67f08e9f42acedd3d6d29512ac446e15f/datasette/utils/init.py#L563-L579