{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-790257263", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 790257263, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDI1NzI2Mw==", "user": {"value": 649467, "label": "mhalle"}, "created_at": "2021-03-04T03:20:23Z", "updated_at": "2021-03-04T03:20:23Z", "author_association": "NONE", "body": "It's kind of an ugly hack, but you can try out what using the fts5 table as an actual datasette-accessible table looks like without changing any datasette code by creating yet another view on top of the fts5 table:\r\n\r\n`create view proxyview as select *, rank, table_fts as fts from table_fts;`\r\n\r\nThat's now visible from datasette, just like any other view, but you can use `fts match escape_fts(search_string) order by rank`.\r\n\r\nThis is only good as a proof of concept because you're inefficiently going from view -> fts5 external content table -> view -> data table. However, it does show it works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-789409126", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 789409126, "node_id": "MDEyOklzc3VlQ29tbWVudDc4OTQwOTEyNg==", "user": {"value": 649467, "label": "mhalle"}, "created_at": "2021-03-03T03:57:15Z", "updated_at": "2021-03-03T03:58:40Z", "author_association": "NONE", "body": "In FTS5, I think doing an FTS search is actually much easier than doing a join against the main table like datasette does now. In fact, FTS5 external content tables provide a transparent interface back to the original table or view.\r\n\r\nHere's what I'm currently doing:\r\n* build a view that joins whatever tables I want and rename the columns to non-joiny names (e.g, `chapter.name AS chapter_name` in the view where needed)\r\n* Create an FTS5 table with `content=\"viewname\"`\r\n* As described in the \"external content tables\" section (https://www.sqlite.org/fts5.html#external_content_tables), sql queries can be made directly to the FTS table, which behind the covers makes select calls to the content table when the content of the original columns are needed.\r\n* In addition, you get \"rank\" and \"bm25()\" available to you when you select on the _fts table.\r\n\r\nUnfortunately, datasette doesn't currently seem happy being coerced into doing a real query on an fts5 table.  This works:\r\n```select col1, col2, col3 from table_fts where coll1=\"value\" and table_fts match escape_fts(\"search term\") order by rank```\r\n\r\nBut this doesn't work in the datasette SQL query interface:\r\n```select col1, col2, col3 from table_fts where coll1=\"value\" and table_fts match escape_fts(:search) order by rank```   (the \"search\" input text field doesn't show up)\r\n\r\nFor what datasette is doing right now, I think you could just use contentless fts5 tables (`content=\"\"`), since all you care about is the rowid since all you're doing a subselect to get the rowid anyway. In fts5, that's just a contentless table.\r\n\r\nI guess if you want to follow this suggestion, you'd need a somewhat different code path for fts5.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-726419027", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 726419027, "node_id": "MDEyOklzc3VlQ29tbWVudDcyNjQxOTAyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-13T00:09:04Z", "updated_at": "2020-11-13T00:09:04Z", "author_association": "OWNER", "body": "Part of the challenge here is that this is the first time the `TableView` will have had a complete rewrite of the SQL it is going to execute. That SQL is currently constructed here: https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/table.py#L628-L636", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-723740546", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 723740546, "node_id": "MDEyOklzc3VlQ29tbWVudDcyMzc0MDU0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-09T04:01:50Z", "updated_at": "2020-11-09T04:01:50Z", "author_association": "OWNER", "body": "I should depend on `sqlite-fts4` - I'm doing that in `sqlite-utils` now and it works great: https://github.com/simonw/sqlite-utils/issues/198", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-721896822", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 721896822, "node_id": "MDEyOklzc3VlQ29tbWVudDcyMTg5NjgyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-04T18:23:29Z", "updated_at": "2020-11-04T18:23:29Z", "author_association": "OWNER", "body": "Worth noting that joining to get the rank works for FTS5 but not for FTS4 - see comment here: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721420539\r\n\r\nEasiest solution would be to only support sort-by-rank for FTS5 tables. Alternative would be to depend on https://github.com/simonw/sqlite-fts4", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-675725464", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 675725464, "node_id": "MDEyOklzc3VlQ29tbWVudDY3NTcyNTQ2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-18T21:18:07Z", "updated_at": "2020-08-18T21:18:35Z", "author_association": "OWNER", "body": "I want this on the table page - but that means that the table page will need to run a slightly more complex query since it needs access to a `rank` column to sort by - which it gets from running a join.\r\n\r\nBUT... that join needs to be constructed in a way that keeps existing filters, `?_where=` clauses etc intact.\r\n\r\nHere's a prototype using SQLite CTEs: https://register-of-members-interests.datasettes.com/regmem?sql=with+original+as+%28select+rowid%2C+*+from+items%29%0D%0Aselect%0D%0A++original.*%2C%0D%0A++items_fts.rank+as+items_fts_rank%0D%0Afrom%0D%0A++original+join+items_fts+on+original.rowid+%3D+items_fts.rowid%0D%0Awhere%0D%0A++items_fts+match+escape_fts%28%3Asearch%29%0D%0Aorder+by+items_fts_rank+desc+limit+10&search=hotel\r\n\r\n```sql\r\nwith original as (\r\n  select\r\n    rowid,\r\n    *\r\n  from\r\n    items\r\n)\r\nselect\r\n  original.*,\r\n  items_fts.rank as items_fts_rank\r\nfrom\r\n  original\r\n  join items_fts on original.rowid = items_fts.rowid\r\nwhere\r\n  items_fts match escape_fts(:search)\r\norder by\r\n  items_fts_rank desc\r\nlimit\r\n  10\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-504880796", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 504880796, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDg4MDc5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-24T06:47:23Z", "updated_at": "2019-06-24T06:47:23Z", "author_association": "OWNER", "body": "I did a bunch of research relevant to this a while ago: https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}