{"html_url": "https://github.com/simonw/datasette/issues/1259#issuecomment-803674728", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1259", "id": 803674728, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY3NDcyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-21T22:55:31Z", "updated_at": "2021-03-21T22:55:31Z", "author_association": "OWNER", "body": "CTEs were added in 2014-02-03 SQLite 3.8.3 - so I think it's OK to depend on them for Datasette.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 830567275, "label": "Research using CTEs for faster facet counts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1259#issuecomment-797827038", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1259", "id": 797827038, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NzgyNzAzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-13T00:15:40Z", "updated_at": "2021-03-13T00:15:40Z", "author_association": "OWNER", "body": "If all of the facets were being calculated in a single query, I'd be willing to bump the facet time limit up to something a lot higher, maybe even a full second. There's a chance that could work amazingly well with a materialized CTE.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 830567275, "label": "Research using CTEs for faster facet counts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1259#issuecomment-797804869", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1259", "id": 797804869, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NzgwNDg2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-12T23:05:05Z", "updated_at": "2021-03-12T23:05:05Z", "author_association": "OWNER", "body": "I wonder if I could optimize facet suggestion in the same way?\r\n\r\nOne challenge: the query time limit will apply to the full CTE query, not to the individual columns.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 830567275, "label": "Research using CTEs for faster facet counts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1259#issuecomment-797801075", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1259", "id": 797801075, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NzgwMTA3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-12T22:53:56Z", "updated_at": "2021-03-12T22:55:16Z", "author_association": "OWNER", "body": "OK, a better comparison:\r\n\r\nhttps://global-power-plants.datasettes.com/global-power-plants?sql=WITH+data+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Acountry_long+as+%28select+%0D%0A++%27country_long%27+as+col%2C+country_long+as+value%2C+count%28*%29+as+c+from+data+group+by+country_long%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aprimary_fuel+as+%28%0D%0Aselect%0D%0A++%27primary_fuel%27+as+col%2C+primary_fuel+as+value%2C+count%28*%29+as+c+from+data+group+by+primary_fuel%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aowner+as+%28%0D%0Aselect%0D%0A++%27owner%27+as+col%2C+owner+as+value%2C+count%28*%29+as+c+from+data+group+by+owner%0D%0A++order+by+c+desc+limit+31%0D%0A%29%0D%0Aselect+*+from+primary_fuel+union+select+*+from+country_long%0D%0Aunion+select+*+from+owner+order+by+col%2C+c+desc calculates facets against three columns. It takes **78.5ms** (and 34.5ms when I refreshed it, presumably after warming some SQLite caches of some sort).\r\n\r\nhttps://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet=country_long&_facet=primary_fuel&_trace=1&_size=0 shows those facets with size=0 on the SQL query - and shows a SQL trace at the bottom of the page.\r\n\r\nThe country_long facet query takes 45.36ms, owner takes 38.45ms, primary_fuel takes 49.04ms - so a total of 132.85ms\r\n\r\nThat's against https://global-power-plants.datasettes.com/-/versions says SQLite 3.27.3 - so even on a SQLite version that doesn't materialize the CTEs there's a significant performance boost to doing all three facets in a single CTE query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 830567275, "label": "Research using CTEs for faster facet counts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1259#issuecomment-797790017", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1259", "id": 797790017, "node_id": "MDEyOklzc3VlQ29tbWVudDc5Nzc5MDAxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-12T22:22:12Z", "updated_at": "2021-03-12T22:22:12Z", "author_association": "OWNER", "body": "https://sqlite.org/lang_with.html\r\n\r\n>  Prior to SQLite 3.35.0, all CTEs where treated as if the NOT MATERIALIZED phrase was present\r\n\r\nIt looks like this optimization is completely unavailable on SQLite prior to 3.35.0 (released 12th March 2021). But I could still rewrite the faceting to work in this way, using the exact same SQL - it would just be significantly faster on 3.35.0+ (assuming it's actually faster in practice - would need to benchmark).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 830567275, "label": "Research using CTEs for faster facet counts"}, "performed_via_github_app": null}