{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-790257263", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 790257263, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDI1NzI2Mw==", "user": {"value": 649467, "label": "mhalle"}, "created_at": "2021-03-04T03:20:23Z", "updated_at": "2021-03-04T03:20:23Z", "author_association": "NONE", "body": "It's kind of an ugly hack, but you can try out what using the fts5 table as an actual datasette-accessible table looks like without changing any datasette code by creating yet another view on top of the fts5 table:\r\n\r\n`create view proxyview as select *, rank, table_fts as fts from table_fts;`\r\n\r\nThat's now visible from datasette, just like any other view, but you can use `fts match escape_fts(search_string) order by rank`.\r\n\r\nThis is only good as a proof of concept because you're inefficiently going from view -> fts5 external content table -> view -> data table. However, it does show it works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-789409126", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 789409126, "node_id": "MDEyOklzc3VlQ29tbWVudDc4OTQwOTEyNg==", "user": {"value": 649467, "label": "mhalle"}, "created_at": "2021-03-03T03:57:15Z", "updated_at": "2021-03-03T03:58:40Z", "author_association": "NONE", "body": "In FTS5, I think doing an FTS search is actually much easier than doing a join against the main table like datasette does now. In fact, FTS5 external content tables provide a transparent interface back to the original table or view.\r\n\r\nHere's what I'm currently doing:\r\n* build a view that joins whatever tables I want and rename the columns to non-joiny names (e.g, `chapter.name AS chapter_name` in the view where needed)\r\n* Create an FTS5 table with `content=\"viewname\"`\r\n* As described in the \"external content tables\" section (https://www.sqlite.org/fts5.html#external_content_tables), sql queries can be made directly to the FTS table, which behind the covers makes select calls to the content table when the content of the original columns are needed.\r\n* In addition, you get \"rank\" and \"bm25()\" available to you when you select on the _fts table.\r\n\r\nUnfortunately, datasette doesn't currently seem happy being coerced into doing a real query on an fts5 table.  This works:\r\n```select col1, col2, col3 from table_fts where coll1=\"value\" and table_fts match escape_fts(\"search term\") order by rank```\r\n\r\nBut this doesn't work in the datasette SQL query interface:\r\n```select col1, col2, col3 from table_fts where coll1=\"value\" and table_fts match escape_fts(:search) order by rank```   (the \"search\" input text field doesn't show up)\r\n\r\nFor what datasette is doing right now, I think you could just use contentless fts5 tables (`content=\"\"`), since all you care about is the rowid since all you're doing a subselect to get the rowid anyway. In fts5, that's just a contentless table.\r\n\r\nI guess if you want to follow this suggestion, you'd need a somewhat different code path for fts5.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/220#issuecomment-783662968", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/220", "id": 783662968, "node_id": "MDEyOklzc3VlQ29tbWVudDc4MzY2Mjk2OA==", "user": {"value": 649467, "label": "mhalle"}, "created_at": "2021-02-22T20:44:51Z", "updated_at": "2021-02-22T20:44:51Z", "author_association": "NONE", "body": "Actually, coming back to this, I have a clearer use case for enabling fts generation for views: making it easier to bring in text from lookup tables and other joins.  \r\n\r\nThe datasette documentation describes populating an fts table like so:\r\n```\r\nINSERT INTO \"items_fts\" (rowid, name, description, category_name)\r\n    SELECT items. rowid,\r\n    items.name,\r\n    items.description,\r\n    categories.name\r\n    FROM items JOIN categories ON items.category_id=categories.id;\r\n```\r\nAlternatively if you have fts support in sqlite_utils for views (which sqlite and fts5 support), you can do the same thing just by creating a view that captures the above joins as columns, then creating an fts table from that view. Such an fts table can be created using sqlite_utils, where one created with your method can't. \r\n\r\nThe resulting fts table can then be used by a whole family of related tables and views in the manner you described earlier in this issue. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 783778672, "label": "Better error message for *_fts methods against views"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/171#issuecomment-714219725", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/171", "id": 714219725, "node_id": "MDEyOklzc3VlQ29tbWVudDcxNDIxOTcyNQ==", "user": {"value": 649467, "label": "mhalle"}, "created_at": "2020-10-22T04:38:35Z", "updated_at": "2020-10-22T04:38:35Z", "author_association": "NONE", "body": "Thanks. As I said, I think the result (being able to query tree structures like ancestors and descendants) is more important than the implementation, and I agree that this particular sqlite extension is too obscure. Just providing an sqlite utility to build or rebuild a transitive closure table might be more generically useful. I find that hierarchical data shows up pretty frequently in some data science problems.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707407567, "label": "Idea: transitive closure tables for tree structures"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1003#issuecomment-706302863", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1003", "id": 706302863, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjMwMjg2Mw==", "user": {"value": 649467, "label": "mhalle"}, "created_at": "2020-10-09T17:17:06Z", "updated_at": "2020-10-09T17:17:06Z", "author_association": "NONE", "body": "I agree on the descriptive and python-consistent naming. There is already a tojson, but frankly i find the \"to\" and \"from\" confusing in a text templating language where what's a string and what's data isn't 100% transparent.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718238967, "label": "from_json jinja2 filter"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/144#issuecomment-346427794", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/144", "id": 346427794, "node_id": "MDEyOklzc3VlQ29tbWVudDM0NjQyNzc5NA==", "user": {"value": 649467, "label": "mhalle"}, "created_at": "2017-11-22T17:55:45Z", "updated_at": "2017-11-22T17:55:45Z", "author_association": "NONE", "body": "Thanks. There is a way to use pip to grab apsw, which also let's you configure it (flags to build extensions, use an internal sqlite, etc). Don't know how that works as a dependency for another package, though.\n\nOn November 22, 2017 11:38:06 AM EST, Simon Willison <notifications@github.com> wrote:\n>I have a solution for FTS already, but I'm interested in apsw as a\n>mechanism for allowing custom virtual tables to be written in Python\n>(pysqlite only lets you write custom functions)\n>\n>Not having PyPI support is pretty tough though. I'm planning a\n>plugin/extension system which would be ideal for things like an\n>optional apsw mode, but that's a lot harder if apsw isn't in PyPI.\n>\n>-- \n>You are receiving this because you authored the thread.\n>Reply to this email directly or view it on GitHub:\n>https://github.com/simonw/datasette/issues/144#issuecomment-346405660\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 276091279, "label": "apsw as alternative sqlite3 binding (for full text search)"}, "performed_via_github_app": null}