{"html_url": "https://github.com/simonw/datasette/issues/1884#issuecomment-1321460293", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1884", "id": 1321460293, "node_id": "IC_kwDOBm6k_c5Ow-JF", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2022-11-21T04:40:55Z", "updated_at": "2022-11-21T04:40:55Z", "author_association": "CONTRIBUTOR", "body": "Counting any virtual tables can be pretty tricky. On one hand, counting a [CSV virtual table](https://www.sqlite.org/csv.html) would return the number of rows in the CSV, which is helpful (but can be I/O intensive). Counting a [FTS5 virtual table](https://www.sqlite.org/fts5.html) would return the number of entries in the FTS index, which is kindof helpful, but can be misleading in some cases.\r\n\r\nOn the other hand, arbitrarily running `COUNT(*)` on some virtual tables can be incredibly expensive. SQLite offers new shortcuts/pushdowns on `COUNT(*)` queries for virtual tables, and instead calls the underlying vtab implementation and iterates through all rows in the table without discretion. For example, a virtual table that's backed by a Postgres table would call `select * from pg_table`, which would use up a lot of network and CPU calls. Or a virtual table backed by a [google sheet](https://github.com/0x6b/libgsqlite) would make network/API requests to get all the rows from the sheet just to make a count.\r\n\r\nThe [`pragma_table_list`](https://www.sqlite.org/pragma.html#pragma_table_list) pragma tells you when a table is a regular table or virtual (in the `type` column), but was only added in version 3.37.0 (2021-11-27). \r\n\r\n\r\nPersonally, I wouldnt try to `COUNT(*)` virtual tables - it depends on how the virtual table is implemented, it requires that the connection has the proper extensions loaded, and it may accientally cause perf issues for new-age extensions. A few extensions that I'm writing have virtual tables that wouldn't benefit much from `COUNT(*)`, and the fact that SQLite iterates through all rows in a table to count just makes things worse. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1439009231, "label": "Exclude virtual tables from datasette inspect"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1884#issuecomment-1314066229", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1884", "id": 1314066229, "node_id": "IC_kwDOBm6k_c5OUw81", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-11-14T16:48:35Z", "updated_at": "2022-11-14T16:48:35Z", "author_association": "CONTRIBUTOR", "body": "I'm realizing I don't know if a virtual table will ever return a count. Maybe it depends on the implementation. For these three, just checking now, it'll always return zero.\r\n\r\nThat said, I'm not sure there's any downside to having them return zero and caching that. (They're hidden, too.) ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1439009231, "label": "Exclude virtual tables from datasette inspect"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1884#issuecomment-1314054300", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1884", "id": 1314054300, "node_id": "IC_kwDOBm6k_c5OUuCc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-14T16:40:06Z", "updated_at": "2022-11-14T16:40:06Z", "author_association": "OWNER", "body": "I wonder if there are any reasons that inspect SHOULD try to count virtual tables? Like are there any likely uses for a cirial table where the count is both interesting and likely to be accessed often enough that it's worth caching?\r\n\r\nI have an issue open to add a setting to disable table counts entirely:\r\n\r\n- #1818 \r\n\r\nMaybe that should be expanded to automatically disable row counts for virtual tables entirely? Which would mean no count would be shown for them in the UI.\r\n\r\nIf you desperately wanted a count you would  then have to run a count(*) query against them explicitly.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1439009231, "label": "Exclude virtual tables from datasette inspect"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1884#issuecomment-1313962183", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1884", "id": 1313962183, "node_id": "IC_kwDOBm6k_c5OUXjH", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-11-14T15:46:32Z", "updated_at": "2022-11-14T15:46:32Z", "author_association": "CONTRIBUTOR", "body": "It does work, though I think it's probably still worth excluding virtual tables that will always be zero. Here's the same inspection as before, now with `--load-extension spatialite`:\r\n\r\n```json\r\n{\r\n  \"alltheplaces\": {\r\n    \"hash\": \"0843cfe414439ab903c22d1121b7ddbc643418c35c7f0edbcec82ef1452411df\",\r\n    \"size\": 963375104,\r\n    \"file\": \"alltheplaces.db\",\r\n    \"tables\": {\r\n      \"spatial_ref_sys\": {\r\n        \"count\": 6215\r\n      },\r\n      \"spatialite_history\": {\r\n        \"count\": 18\r\n      },\r\n      \"sqlite_sequence\": {\r\n        \"count\": 2\r\n      },\r\n      \"geometry_columns\": {\r\n        \"count\": 3\r\n      },\r\n      \"spatial_ref_sys_aux\": {\r\n        \"count\": 6164\r\n      },\r\n      \"views_geometry_columns\": {\r\n        \"count\": 0\r\n      },\r\n      \"virts_geometry_columns\": {\r\n        \"count\": 0\r\n      },\r\n      \"geometry_columns_statistics\": {\r\n        \"count\": 3\r\n      },\r\n      \"views_geometry_columns_statistics\": {\r\n        \"count\": 0\r\n      },\r\n      \"virts_geometry_columns_statistics\": {\r\n        \"count\": 0\r\n      },\r\n      \"geometry_columns_field_infos\": {\r\n        \"count\": 0\r\n      },\r\n      \"views_geometry_columns_field_infos\": {\r\n        \"count\": 0\r\n      },\r\n      \"virts_geometry_columns_field_infos\": {\r\n        \"count\": 0\r\n      },\r\n      \"geometry_columns_time\": {\r\n        \"count\": 3\r\n      },\r\n      \"geometry_columns_auth\": {\r\n        \"count\": 3\r\n      },\r\n      \"views_geometry_columns_auth\": {\r\n        \"count\": 0\r\n      },\r\n      \"virts_geometry_columns_auth\": {\r\n        \"count\": 0\r\n      },\r\n      \"data_licenses\": {\r\n        \"count\": 10\r\n      },\r\n      \"sql_statements_log\": {\r\n        \"count\": 0\r\n      },\r\n      \"states\": {\r\n        \"count\": 56\r\n      },\r\n      \"counties\": {\r\n        \"count\": 3234\r\n      },\r\n      \"idx_states_geometry_rowid\": {\r\n        \"count\": 56\r\n      },\r\n      \"idx_states_geometry_node\": {\r\n        \"count\": 3\r\n      },\r\n      \"idx_states_geometry_parent\": {\r\n        \"count\": 2\r\n      },\r\n      \"idx_counties_geometry_rowid\": {\r\n        \"count\": 3234\r\n      },\r\n      \"idx_counties_geometry_node\": {\r\n        \"count\": 98\r\n      },\r\n      \"idx_counties_geometry_parent\": {\r\n        \"count\": 97\r\n      },\r\n      \"idx_places_geometry_rowid\": {\r\n        \"count\": 1236796\r\n      },\r\n      \"idx_places_geometry_node\": {\r\n        \"count\": 38163\r\n      },\r\n      \"idx_places_geometry_parent\": {\r\n        \"count\": 38162\r\n      },\r\n      \"places\": {\r\n        \"count\": 1332609\r\n      },\r\n      \"SpatialIndex\": {\r\n        \"count\": 0\r\n      },\r\n      \"ElementaryGeometries\": {\r\n        \"count\": 0\r\n      },\r\n      \"KNN\": {\r\n        \"count\": 0\r\n      },\r\n      \"idx_states_geometry\": {\r\n        \"count\": 56\r\n      },\r\n      \"idx_counties_geometry\": {\r\n        \"count\": 3234\r\n      },\r\n      \"idx_places_geometry\": {\r\n        \"count\": 1236796\r\n      }\r\n    }\r\n  }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1439009231, "label": "Exclude virtual tables from datasette inspect"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1884#issuecomment-1311269045", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1884", "id": 1311269045, "node_id": "IC_kwDOBm6k_c5OKGC1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-11T06:08:28Z", "updated_at": "2022-11-11T06:08:28Z", "author_association": "OWNER", "body": "Does that work if you add `--load-extension spatialite`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1439009231, "label": "Exclude virtual tables from datasette inspect"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1884#issuecomment-1309735529", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1884", "id": 1309735529, "node_id": "IC_kwDOBm6k_c5OEPpp", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-11-10T03:57:23Z", "updated_at": "2022-11-10T03:57:23Z", "author_association": "CONTRIBUTOR", "body": "Here's how to get a list of virtual tables: https://stackoverflow.com/questions/46617118/how-to-fetch-names-of-virtual-tables", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1439009231, "label": "Exclude virtual tables from datasette inspect"}, "performed_via_github_app": null}