{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747754082", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747754082, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzc1NDA4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-17T23:04:13Z", "updated_at": "2020-12-17T23:04:13Z", "author_association": "OWNER", "body": "Pages that need a list of all databases - the index page and /-/databases for example - could trigger a \"check for new directories in the configured directories\" scan.\r\n\r\nThat scan would run at most once every 5 (n) seconds - the check is triggered if it\u2019s run more recently than that it doesn\u2019t run.\r\n\r\nHopefully this means it could be done as a blocking operation, rather than trying to run it in a thread.\r\n\r\nWhen it runs it scans for *.db or *.sqlite files (maybe one or two other extensions) that it hasn\u2019t seen before. It also checks that the existing list of known database files still exists.\r\n\r\nIf it finds any new ones it connects to them once to run `.schema`. It also runs `PRAGMA schema_version` on each known database so that it can compare the schema version number to the last one it saw. That's how it detects if there are new tables or if the cached schema needs to be updated.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747754229", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747754229, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzc1NDIyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-17T23:04:38Z", "updated_at": "2020-12-17T23:04:38Z", "author_association": "OWNER", "body": "Open question: will this work for hundreds of database files, or is the overhead of connecting to each of 100 databases in turn to run `PRAGMA schema_version` too high?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747764712", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747764712, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzc2NDcxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-17T23:16:31Z", "updated_at": "2020-12-17T23:16:31Z", "author_association": "OWNER", "body": "Quick micro-benchmark, run against a folder with 46 database files adding up to 1.4GB total:\r\n```python\r\nimport pathlib, sqlite3, time\r\n\r\npaths = list(pathlib.Path(\".\").glob('*.db'))\r\n\r\ndef schema_version(path):\r\n    db = sqlite3.connect(path)\r\n    version = db.execute(\"PRAGMA schema_version\").fetchall()[0]\r\n    db.close()\r\n    return version\r\n\r\ndef all():\r\n    versions = {}\r\n    for path in paths:\r\n        versions[path.name] = schema_version(path)\r\n    return versions\r\n\r\nstart = time.time(); all(); print(time.time() - start)\r\n# 0.012346982955932617\r\n```\r\nSo that's 12ms.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747766310", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747766310, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzc2NjMxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-17T23:20:49Z", "updated_at": "2020-12-17T23:20:49Z", "author_association": "OWNER", "body": "I tried against my entire `~/Development/Dropbox` folder - deeply nested with 381 SQLite database files in sub-folders - and it took 25s! But it turned out 23.9s of that was the call to `pathlib.Path(\"/Users/simon/Dropbox/Development\").glob('**/*.db')`.\r\n\r\nSo it looks like connecting to a SQLite database file and getting the schema version is extremely fast. Scanning directories is slower.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747767055", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747767055, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzc2NzA1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-17T23:22:41Z", "updated_at": "2020-12-17T23:22:41Z", "author_association": "OWNER", "body": "It's just recursion that's expensive. I created 380 empty SQLite databases in a folder and timed `list(pathlib.Path(\"/tmp\").glob(\"*.db\"));` and it took 0.002s.\r\n\r\nSo maybe I tell users that all SQLite databases have to be in the root folder.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747767499", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747767499, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzc2NzQ5OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-17T23:23:44Z", "updated_at": "2020-12-17T23:23:44Z", "author_association": "OWNER", "body": "Grabbing the schema version of 380 files in the root directory takes 70ms.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747767598", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747767598, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzc2NzU5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-17T23:24:03Z", "updated_at": "2020-12-17T23:24:03Z", "author_association": "OWNER", "body": "I'm going to assume that even the heaviest user will have trouble going beyond a few hundred database files, so this is fine.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747768112", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747768112, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzc2ODExMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-17T23:25:21Z", "updated_at": "2020-12-17T23:25:21Z", "author_association": "OWNER", "body": "Next challenge: figure out how to use the `Database` class from https://github.com/simonw/datasette/blob/0.53/datasette/database.py for an in-memory database which persists data for the duration of the lifetime of the server, and allows access to that in-memory database from multiple threads in a way that lets them see each other's changes.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747803268", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747803268, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgwMzI2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T01:08:40Z", "updated_at": "2020-12-18T01:08:40Z", "author_association": "OWNER", "body": "Next step: design a schema for the in-memory database table that exposes all of the tables.\r\n\r\nI want to support things like:\r\n\r\n- Show me all of the tables\r\n- Show me the columns in a table\r\n- Show me all tables that contain a `tags` column\r\n- Show me the indexes\r\n- Show me every table configured for full-text search\r\n\r\nMaybe a starting point would be to build concrete tables using the results of things like `PRAGMA foreign_key_list(table)` and `PRAGMA table_xinfo(table)` - note though that `table_xinfo` is SQLite 3.26.0 or higher, as shown here: https://github.com/simonw/datasette/blob/5e9895c67f08e9f42acedd3d6d29512ac446e15f/datasette/utils/__init__.py#L563-L579", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747804254", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747804254, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgwNDI1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T01:12:13Z", "updated_at": "2020-12-18T01:12:13Z", "author_association": "OWNER", "body": "Prototype: https://latest.datasette.io/fixtures?sql=select+%27facetable%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27facetable%27%29%0D%0Aunion%0D%0Aselect+%27searchable%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27searchable%27%29%0D%0Aunion%0D%0Aselect+%27compound_three_primary_keys%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27compound_three_primary_keys%27%29\r\n\r\n```sql\r\nselect 'facetable' as 'table', * from pragma_table_xinfo('facetable')\r\nunion\r\nselect 'searchable' as 'table', * from pragma_table_xinfo('searchable')\r\nunion\r\nselect 'compound_three_primary_keys' as 'table', * from pragma_table_xinfo('compound_three_primary_keys')\r\n```\r\n\r\n\r\ntable | cid | name | type | notnull | dflt_value | pk | hidden\r\n-- | -- | -- | -- | -- | -- | -- | --\r\ncompound_three_primary_keys | 0 | pk1 | varchar(30) | 0 | \u00a0 | 1 | 0\r\ncompound_three_primary_keys | 1 | pk2 | varchar(30) | 0 | \u00a0 | 2 | 0\r\ncompound_three_primary_keys | 2 | pk3 | varchar(30) | 0 | \u00a0 | 3 | 0\r\ncompound_three_primary_keys | 3 | content | text | 0 | \u00a0 | 0 | 0\r\nfacetable | 0 | pk | integer | 0 | \u00a0 | 1 | 0\r\nfacetable | 1 | created | text | 0 | \u00a0 | 0 | 0\r\nfacetable | 2 | planet_int | integer | 0 | \u00a0 | 0 | 0\r\nfacetable | 3 | on_earth | integer | 0 | \u00a0 | 0 | 0\r\nfacetable | 4 | state | text | 0 | \u00a0 | 0 | 0\r\nfacetable | 5 | city_id | integer | 0 | \u00a0 | 0 | 0\r\nfacetable | 6 | neighborhood | text | 0 | \u00a0 | 0 | 0\r\nfacetable | 7 | tags | text | 0 | \u00a0 | 0 | 0\r\nfacetable | 8 | complex_array | text | 0 | \u00a0 | 0 | 0\r\nfacetable | 9 | distinct_some_null | \u00a0 | 0 | \u00a0 | 0 | 0\r\nsearchable | 0 | pk | integer | 0 | \u00a0 | 1 | 0\r\nsearchable | 1 | text1 | text | 0 | \u00a0 | 0 | 0\r\nsearchable | 2 | text2 | text | 0 | \u00a0 | 0 | 0\r\nsearchable | 3 | name with . and spaces | text | 0 | \u00a0 | 0 | 0", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747805275", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747805275, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgwNTI3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T01:15:27Z", "updated_at": "2020-12-18T01:16:17Z", "author_association": "OWNER", "body": "This query uses a join to pull foreign key information for every table: https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%0D%0Aselect%0D%0A++tables.name+as+%27table%27%2C%0D%0A++foo.*%0D%0Afrom%0D%0A++tables%0D%0A++join+pragma_foreign_key_list%28tables.name%29+foo\r\n\r\n```sql\r\nwith tables as (\r\n  select\r\n    name\r\n  from\r\n    sqlite_master\r\n  where\r\n    type = 'table'\r\n)\r\nselect\r\n  tables.name as 'table',\r\n  foo.*\r\nfrom\r\n  tables\r\n  join pragma_foreign_key_list(tables.name) foo\r\n```\r\n\r\nSame query for `pragma_table_xinfo`: https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%0D%0Aselect%0D%0A++tables.name+as+%27table%27%2C%0D%0A++foo.*%0D%0Afrom%0D%0A++tables%0D%0A++join+pragma_table_xinfo%28tables.name%29+foo", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747807289", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747807289, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgwNzI4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T01:22:05Z", "updated_at": "2020-12-18T01:22:05Z", "author_association": "OWNER", "body": "Here's a simpler query pattern (not using CTEs so should work on older versions of SQLite) - this one lists all indexes for all tables:\r\n```sql\r\nselect\r\n  sqlite_master.name as 'table',\r\n  indexes.*\r\nfrom\r\n  sqlite_master\r\n  join pragma_index_list(sqlite_master.name) indexes\r\nwhere\r\n  sqlite_master.type = 'table'\r\n```\r\nhttps://latest.datasette.io/fixtures?sql=select%0D%0A++sqlite_master.name+as+%27table%27%2C%0D%0A++indexes.*%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_index_list%28sqlite_master.name%29+indexes%0D%0Awhere%0D%0A++sqlite_master.type+%3D+%27table%27", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747807891", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747807891, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgwNzg5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T01:23:59Z", "updated_at": "2020-12-18T01:23:59Z", "author_association": "OWNER", "body": "https://www.sqlite.org/pragma.html#pragfunc says:\r\n>  * This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. \r\n>  * The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747809670", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747809670, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgwOTY3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T01:29:30Z", "updated_at": "2020-12-18T01:29:30Z", "author_association": "OWNER", "body": "I've been rediscovering the pattern I already documented in this TIL: https://github.com/simonw/til/blob/main/sqlite/list-all-columns-in-a-database.md#better-alternative-using-a-join", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747833639", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747833639, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgzMzYzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T02:49:40Z", "updated_at": "2020-12-18T03:52:12Z", "author_association": "OWNER", "body": "I'm going to use five tables to start off with:\r\n\r\n- `databases` - a list of databases. Each one has a `name`, `path` (if it's on disk), `is_memory`, `schema_version`\r\n- `tables` - a list of tables. Each row is `database_name`, `table_name`, `sql` (the create table statement) - may add more tables in the future, in particular maybe a `last_row_count` to cache results of counting the rows.\r\n- `columns` - a list of columns. It's the output of `pragma_table_xinfo` with the `database_name` and `table_name` columns added at the beginning.\r\n- `foreign_keys` - a list of foreign keys - `pragma_foreign_key_list` output plus `database_name` and `table_name`.\r\n- `indexes` - a list of indexes - `pragma_table_xinfo` output plus `database_name` and `table_name`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747834113", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747834113, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgzNDExMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T02:51:13Z", "updated_at": "2020-12-18T02:51:20Z", "author_association": "OWNER", "body": "SQLite uses `indexes` rather than `indices` as the plural, so I'll go with that: https://sqlite.org/lang_createindex.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747834462", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747834462, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgzNDQ2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T02:52:19Z", "updated_at": "2020-12-18T02:52:26Z", "author_association": "OWNER", "body": "Maintaining this database will be the responsibility of a subclass of `Database` called `_SchemaDatabase` which will be managed by the `Datasette` instance.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747834762", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747834762, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzgzNDc2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T02:53:22Z", "updated_at": "2020-12-18T02:53:22Z", "author_association": "OWNER", "body": "I think I'm going to have to build this without using the `pragma_x()` SQL functions as they were only added in 3.16 in 2017-01-02 and I've seen plenty of Datasette instances running on older versions of SQLite.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747847180", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747847180, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg0NzE4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T03:35:15Z", "updated_at": "2020-12-18T03:35:15Z", "author_association": "OWNER", "body": "Simpler implementation idea: a Datasette method `.refresh_schemas()` which loops through all known databases, checks their schema version and updates the in-memory schemas database if they have changed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747847405", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747847405, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg0NzQwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T03:36:04Z", "updated_at": "2020-12-18T03:36:04Z", "author_association": "OWNER", "body": "I could have another table that stores the combined rows from `sqlite_m\u00e1ster` on every connected database so I have a copy of the schema SQL.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747861357", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747861357, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg2MTM1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T04:32:52Z", "updated_at": "2020-12-18T04:32:52Z", "author_association": "OWNER", "body": "I need to figure out how this will interact with Datasette permissions.\r\n\r\nIf some tables are private, but others are public, should users be able to see the private tables listed in the schema metadata?\r\n\r\nIf not, how can that mechanism work?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747861556", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747861556, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg2MTU1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T04:33:45Z", "updated_at": "2020-12-18T04:33:45Z", "author_association": "OWNER", "body": "One solution on permissions: if Datasette had an efficient way of saying \"list the tables that this user has access to\" I could use that as a filter any time the user views the schema information. The implementation could be tricky though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747862001", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747862001, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg2MjAwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T04:35:34Z", "updated_at": "2020-12-18T04:35:34Z", "author_association": "OWNER", "body": "I do need to solve the permissions problem properly though, because one of the goals of this system is to provide a paginated, searchable list of databases and tables for the homepage of the instance - #991.\r\n\r\nAs such, the homepage will need to be able to display only the tables and databases that the user has permission to view.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747864080", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747864080, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg2NDA4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T04:43:29Z", "updated_at": "2020-12-18T04:43:29Z", "author_association": "OWNER", "body": "I may be overthinking that problem. Many queries are fast in SQLite. If a Datasette instance has 1,000 connected tables will even that be a performance problem for permission checks? I should benchmark to find out.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747864831", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747864831, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg2NDgzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T04:46:18Z", "updated_at": "2020-12-18T04:46:18Z", "author_association": "OWNER", "body": "The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75\r\n\r\nA paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-747893704", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 747893704, "node_id": "MDEyOklzc3VlQ29tbWVudDc0Nzg5MzcwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T06:19:13Z", "updated_at": "2020-12-18T06:19:13Z", "author_association": "OWNER", "body": "I'm not going to block this issue on permissions - I will tackle the efficient bulk permissions problem in #1152.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-748260118", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 748260118, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODI2MDExOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T18:54:12Z", "updated_at": "2020-12-18T18:54:12Z", "author_association": "OWNER", "body": "I'm going to tidy this up and land it. A couple of additional decisions:\r\n\r\n- The database will be called `/_schemas`\r\n- By default it will only be visible to `root` - thus avoiding having to solve the permissions problem with regards to users seeing schemas for tables that are otherwise invisible to them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-748260875", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 748260875, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODI2MDg3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T18:55:12Z", "updated_at": "2020-12-18T18:55:12Z", "author_association": "OWNER", "body": "I'm going to move the code into a `utils/schemas.py` module, to avoid further extending the `Datasette` class definition and to make it more easily testable.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-748351350", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 748351350, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM1MTM1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T22:32:13Z", "updated_at": "2020-12-18T22:32:13Z", "author_association": "OWNER", "body": "Getting all the tests to pass is tricky because this adds a whole extra database to Datasette - and there's various code that loops through `ds.databases` as part of the tests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-748352106", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 748352106, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM1MjEwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T22:34:40Z", "updated_at": "2020-12-18T22:34:40Z", "author_association": "OWNER", "body": "Needs documentation, but I can wait to write that until I've tested out the feature a bit more.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-748354841", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 748354841, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODM1NDg0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-18T22:43:49Z", "updated_at": "2020-12-18T22:43:49Z", "author_association": "OWNER", "body": "For a demo, visit https://latest.datasette.io/login-as-root and then hit https://latest.datasette.io/_schemas", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1150#issuecomment-751476406", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1150", "id": 751476406, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTQ3NjQwNg==", "user": {"value": 18221871, "label": "noklam"}, "created_at": "2020-12-27T14:51:39Z", "updated_at": "2020-12-27T14:51:39Z", "author_association": "NONE", "body": "I like the idea of _internal, it's a nice way to get a data catalog quickly. I wonder if this trick applies to db other than SQLite.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770436876, "label": "Maintain an in-memory SQLite table of connected databases and their tables"}, "performed_via_github_app": null}