issue_comments

This is now possible, for both FTS4 and FTS5 - see #197.

Also that order looks incorrect. It looks like most relevant came back last, not first.

This is a bit surprising: (sqlite-utils) sqlite-utils % sqlite-utils search 24ways-fts4.db articles maps -c title [{"rowid": 41, "title": "What Is Vagrant and Why Should I Care?", "rank": -1.9252039178908076}, {"rowid": 298, "title": "First Steps in VR", "rank": -1.9945466378736434}, {"rowid": 43, "title": "Content Production Planning", "rank": -2.1928058363046143}, {"rowid": 100, "title": "Moo'y Christmas", "rank": -2.2698482999851675}, {"rowid": 91, "title": "Infinite Canvas: Moving Beyond the Page", "rank": -2.290928999035195}, {"rowid": 175, "title": "Front-End Code Reusability with CSS and JavaScript", "rank": -2.498731782924352}, {"rowid": 209, "title": "Feeding the Audio Graph", "rank": -2.619968930100356}, {"rowid": 296, "title": "Animation in Design Systems", "rank": -2.62060151817201}, {"rowid": 118, "title": "Ghosts On The Internet", "rank": -2.7224894534521087}, {"rowid": 77, "title": "Colour Accessibility", "rank": -2.7389782859427343}, {"rowid": 245, "title": "Web Content Accessibility Guidelines 2.1\u2014for People Who Haven\u2019t Read the Update", "rank": -2.9750992611162888}, {"rowid": 56, "title": "Helping VIPs Care About Performance", "rank": -3.0819662908932535}, {"rowid": 109, "title": "Geotag Everywhere with Fire Eagle", "rank": -3.1371975973877277}, {"rowid": 203, "title": "Jobs-to-Be-Done in Your UX Toolbox", "rank": -3.2416719461682733}, {"rowid": 276, "title": "Your jQuery: Now With 67% Less Suck", "rank": -3.4947916564653028}, {"rowid": 58, "title": "Beyond the Style Guide", "rank": -3.7508321464447905}, {"rowid": 225, "title": "Good Ideas Grow on Paper", "rank": -4.120077674716844}, {"rowid": 168, "title": "Unobtrusively Mapping Microformats with jQuery", "rank": -4.662224207228984}, {"rowid": 27, "title": "Putting Design on the Map", "rank": -5.667327088267961}, {"rowid": 220, "title": "Finding Your Way with Static Maps", "rank": -9.952534352591737}] I requested just -c title but also got back rowid and rank.

The breaking changes will be:

• table.search() now returns a generator that produces dictionaries, similar to table.rows_where()
• The -c shortcut no longer works, use --csv instead.
• The -f shortcut no longer works, use --fmt instead.

Still need to update docs.

Thanks to #198 (introducing a rank_bm25() custom function for FTS4) this feature will be able to offer relevance search for both FTS5 AND FTS4 tables.

Having .search() return tuples when .rows_where() returns dictionaries just feels like bad API design to me - it's inconsistent.

Something like this: @db.register_function(replace=True) def my_function(a): return a.upper() If replace=True then this function will be registered even if a my_function of arity 1 has already been registered previously.

It defaults to False though which means the Database object tracks what functions and arities have been registered in the past and silently ignores any new attempts to register the same name/arity.

Should I register the custom rank_bm25 SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed.

Here's a demo of that rank query. I had to sort by rank instead of rank desc - need to double-check that:

https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=with+original+as+(%0D%0A++++select%0D%0A++++++++rowid%2C%0D%0A++++++++%0D%0A++++from+[articles]%0D%0A)%0D%0Aselect%0D%0A++++original.%2C%0D%0A++++rank_bm25(matchinfo([articles_fts]%2C+%27pcnalx%27))+as+rank%0D%0Afrom%0D%0A++++[original]%0D%0A++++join+[articles_fts]+on+[original].rowid+%3D+[articles_fts].rowid%0D%0Awhere%0D%0A++++[articles_fts]+match+%3Aquery%0D%0Aorder+by%0D%0A++++rank%0D%0Alimit+20&query=jquery+maps

Here's the FTS5 query: sql with original as ( select rowid, * from [global-power-plants] ) select original.*, [global-power-plants_fts].rank as rank from [original] join [global-power-plants_fts] on [original].rowid = [global-power-plants_fts].rowid where [global-power-plants_fts] match :query order by rank desc limit 20 The equivalent using rank_bm25() for FTS4 would be: sql with original as ( select rowid, * from [global-power-plants] ) select original.*, rank_bm25(matchinfo([global-power-plants_fts], 'pcnalx')) as rank from [original] join [global-power-plants_fts] on [original].rowid = [global-power-plants_fts].rowid where [global-power-plants_fts] match :query order by rank desc limit 20

I released a 1.0 (and 1.0.1) version of https://github.com/simonw/sqlite-fts4

Example query (from the tests): sql select c0, c1, rank_bm25(matchinfo(search, 'pcnalx')) as bm25 from search where search match ?

I'd have to copy almost all of the code in https://github.com/simonw/sqlite-fts4/blob/master/sqlite_fts4/init.py so I think I will add it as a dependency instead.