{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1029402837", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1029402837, "node_id": "IC_kwDOCGYnMM49W3DV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-03T21:07:35Z", "updated_at": "2022-02-03T21:07:35Z", "author_association": "OWNER", "body": "Closing this - it was something I was curious about, but evidently not curious enough to actually do the work!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008232075", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008232075, "node_id": "IC_kwDOCGYnMM48GGaL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T05:13:15Z", "updated_at": "2022-01-09T05:13:56Z", "author_association": "OWNER", "body": "I think the query that will help solve this is:\r\n\r\n`explain query plan select * from ny_times_us_counties where state = 1 and county = 2`\r\n\r\nIn this case, the query planner needs to decide if it should use the index for the `state` column or the index for the `county` column. That's where the statistics come into play. In particular:\r\n\r\n| tbl | idx | stat |\r\n|----------------------|---------------------------------|---------------|\r\n| ny_times_us_counties | idx_ny_times_us_counties_date | 2092871 2915 |\r\n| ny_times_us_counties | idx_ny_times_us_counties_fips | 2092871 651 |\r\n| ny_times_us_counties | idx_ny_times_us_counties_county | 2092871 1085 |\r\n| ny_times_us_counties | idx_ny_times_us_counties_state | 2092871 37373 |\r\n\r\nThose numbers are explained by this comment in the SQLite C code: https://github.com/sqlite/sqlite/blob/5622c7f97106314719740098cf0854e7eaa81802/src/analyze.c#L41-L55\r\n\r\n```\r\n** There is normally one row per index, with the index identified by the\r\n** name in the idx column. The tbl column is the name of the table to\r\n** which the index belongs. In each such row, the stat column will be\r\n** a string consisting of a list of integers. The first integer in this\r\n** list is the number of rows in the index. (This is the same as the\r\n** number of rows in the table, except for partial indices.) The second\r\n** integer is the average number of rows in the index that have the same\r\n** value in the first column of the index.\r\n```\r\nSo that table is telling us that using a value in the `county` column will filter down to an average of 1,085 rows, whereas filtering on the `state` column will filter down to an average of 37,373 - so clearly the `county` index is the better index to use here!\r\n\r\nJust one catch: against both my` covid.db` and my `covid-analyzed.db` databases the `county` index is picked for both of them - so SQLite is somehow guessing that `county` is a better index even though it doesn't have statistics for that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008229341", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008229341, "node_id": "IC_kwDOCGYnMM48GFvd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:45:38Z", "updated_at": "2022-01-09T04:47:11Z", "author_association": "OWNER", "body": "This is probably too fancy. I think maybe the way to do this is with `select * from [global-power-plants] where \"country_long\" = 'United Kingdom'` - then mess around with stats to see if I can get it to use the index or not based on them.\r\n\r\nHere's the explain for that: https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+*+from+[global-power-plants]+where+%22country_long%22+%3D+%27United+Kingdom%27", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008227625", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008227625, "node_id": "IC_kwDOCGYnMM48GFUp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:25:38Z", "updated_at": "2022-01-09T04:25:38Z", "author_association": "OWNER", "body": "```sql\r\nEXPLAIN QUERY PLAN select country_long, count(*) from [global-power-plants] group by country_long\r\n```\r\nhttps://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+country_long%2C+count%28*%29+from+%5Bglobal-power-plants%5D+group+by+country_long\r\n\r\n> SCAN TABLE global-power-plants USING COVERING INDEX \"global-power-plants_country_long\"", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226862", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008226862, "node_id": "IC_kwDOCGYnMM48GFIu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:17:55Z", "updated_at": "2022-01-09T04:17:55Z", "author_association": "OWNER", "body": "There are some clues as to what effect ANALYZE has in https://www.sqlite.org/optoverview.html\r\n\r\nSome quotes:\r\n\r\n> SQLite might use a skip-scan on an index if it knows that the first one or more columns contain many duplication values. If there are too few duplicates in the left-most columns of the index, then it would be faster to simply step ahead to the next value, and thus do a full table scan, than to do a binary search on an index to locate the next left-column value.\r\n>\r\n> The only way that SQLite can know that there are many duplicates in the left-most columns of an index is if the ANALYZE command has been run on the database. Without the results of ANALYZE, SQLite has to guess at the \"shape\" of the data in the table, and the default guess is that there are an average of 10 duplicates for every value in the left-most column of the index. Skip-scan only becomes profitable (it only gets to be faster than a full table scan) when the number of duplicates is about 18 or more. Hence, a skip-scan is never used on a database that has not been analyzed. \r\n\r\nAnd\r\n\r\n> Join reordering is automatic and usually works well enough that programmers do not have to think about it, especially if ANALYZE has been used to gather statistics about the available indexes, though occasionally some hints from the programmer are needed.\r\n\r\nAnd\r\n\r\n> The various sqlite_statN tables contain information on how selective the various indexes are. For example, the sqlite_stat1 table might indicate that an equality constraint on column x reduces the search space to 10 rows on average, whereas an equality constraint on column y reduces the search space to 3 rows on average. In that case, SQLite would prefer to use index ex2i2 since that index is more selective. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226487", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008226487, "node_id": "IC_kwDOCGYnMM48GFC3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:14:05Z", "updated_at": "2022-01-09T04:14:05Z", "author_association": "OWNER", "body": "Didn't manage to spot a meaningful difference with that database either:\r\n```\r\nanalyze % python3 -m timeit '__import__(\"sqlite3\").connect(\"covid.db\").execute(\"select fips, count(*) from [ny_times_us_counties] group by fips\").fetchall()' \r\n2 loops, best of 5: 101 msec per loop\r\nanalyze % python3 -m timeit '__import__(\"sqlite3\").connect(\"covid-analyzed.db\").execute(\"select fips, count(*) from [ny_times_us_counties] group by fips\").fetchall()'\r\n2 loops, best of 5: 103 msec per loop\r\n```\r\nMaybe `select fips, count(*) from [ny_times_us_counties] group by fips` isn't a good query for testing this?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008220270", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008220270, "node_id": "IC_kwDOCGYnMM48GDhu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T03:12:38Z", "updated_at": "2022-01-09T03:13:15Z", "author_association": "OWNER", "body": "Basically no difference using this very basic benchmark:\r\n```\r\nanalyze % python3 -m timeit '__import__(\"sqlite3\").connect(\"global-power-plants.db\").execute(\"select country_long, count(*) from [global-power-plants] group by country_long\").fetchall()'\r\n100 loops, best of 5: 2.39 msec per loop\r\nanalyze % python3 -m timeit '__import__(\"sqlite3\").connect(\"global-power-plants-analyzed.db\").execute(\"select country_long, count(*) from [global-power-plants] group by country_long\").fetchall()'\r\n100 loops, best of 5: 2.38 msec per loop\r\n```\r\nI should try this against a much larger database.\r\n\r\nhttps://covid-19.datasettes.com/covid.db is 879MB.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219844", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008219844, "node_id": "IC_kwDOCGYnMM48GDbE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T03:08:09Z", "updated_at": "2022-01-09T03:08:09Z", "author_association": "OWNER", "body": "```\r\nanalyze % sqlite-utils global-power-plants-analyzed.db 'analyze'\r\n[{\"rows_affected\": -1}]\r\nanalyze % sqlite-utils tables global-power-plants-analyzed.db \r\n[{\"table\": \"global-power-plants\"},\r\n {\"table\": \"global-power-plants_fts\"},\r\n {\"table\": \"global-power-plants_fts_data\"},\r\n {\"table\": \"global-power-plants_fts_idx\"},\r\n {\"table\": \"global-power-plants_fts_docsize\"},\r\n {\"table\": \"global-power-plants_fts_config\"},\r\n {\"table\": \"sqlite_stat1\"}]\r\nanalyze % sqlite-utils rows global-power-plants-analyzed.db sqlite_stat1 -t\r\ntbl idx stat\r\n------------------------------- ---------------------------------- ---------\r\nglobal-power-plants_fts_config global-power-plants_fts_config 1 1\r\nglobal-power-plants_fts_docsize 33643\r\nglobal-power-plants_fts_idx global-power-plants_fts_idx 199 40 1\r\nglobal-power-plants_fts_data 136\r\nglobal-power-plants \"global-power-plants_owner\" 33643 4\r\nglobal-power-plants \"global-power-plants_country_long\" 33643 202\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219588", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008219588, "node_id": "IC_kwDOCGYnMM48GDXE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T03:06:42Z", "updated_at": "2022-01-09T03:06:42Z", "author_association": "OWNER", "body": "```\r\nanalyze % sqlite-utils indexes global-power-plants.db -t \r\ntable index_name seqno cid name desc coll key\r\n------------------------------ ------------------------------------------------- ------- ----- ------------ ------ ------ -----\r\nglobal-power-plants \"global-power-plants_owner\" 0 12 owner 0 BINARY 1\r\nglobal-power-plants \"global-power-plants_country_long\" 0 1 country_long 0 BINARY 1\r\nglobal-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 0 0 segid 0 BINARY 1\r\nglobal-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 1 1 term 0 BINARY 1\r\nglobal-power-plants_fts_config sqlite_autoindex_global-power-plants_fts_config_1 0 0 k 0 BINARY 1\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219484", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008219484, "node_id": "IC_kwDOCGYnMM48GDVc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T03:05:44Z", "updated_at": "2022-01-09T03:05:44Z", "author_association": "OWNER", "body": "I'll start by running some experiments against the 11MB database file from https://global-power-plants.datasettes.com/global-power-plants.db", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219191", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008219191, "node_id": "IC_kwDOCGYnMM48GDQ3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T03:03:53Z", "updated_at": "2022-01-09T03:03:53Z", "author_association": "OWNER", "body": "Refs:\r\n- #366\r\n- #365", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null}