issues

Table actions

7 rows where milestone = 6079500 and repo = 140912432 sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: user, comments, author_association, created_at (date), updated_at (date), closed_at (date)

type 1

state 1

repo 1

  • sqlite-utils · 7 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
735650864 MDU6SXNzdWU3MzU2NTA4NjQ= 194 3.0 release with some minor breaking changes simonw 9599 closed 0   3.0 6079500 3 2020-11-03T21:36:31Z 2020-11-08T17:19:35Z 2020-11-08T17:19:34Z OWNER  

While working on search (#192) I've spotted a few small changes I would like to make that would break backwards compatibility in minor ways, hence requiring a 3.x release.

db[table].search() - I would like this to default to sorting by rank

Also I'd like to free up the -c and -f options for other purposes from the standard output formats here:

https://github.com/simonw/sqlite-utils/blob/43eae8b193d362f2b292df73e087ed6f10838144/sqlite_utils/cli.py#L48-L58

I'd like -f to be used to indicate a full-text search column during an insert and -c to indicate a column (so you can specify which columns you want to output).

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/194/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
737153927 MDU6SXNzdWU3MzcxNTM5Mjc= 197 Rethink how table.search() method works simonw 9599 closed 0   3.0 6079500 5 2020-11-05T18:04:34Z 2020-11-08T17:07:37Z 2020-11-08T17:07:37Z OWNER  

I need to improve this method to help build sqlite-utils search in #192 (PR is #195).

The challenge is deciding how it should handle sorting by relevance - especially since that is easy in FTS5 but not at all easy in FTS4.

Latest test failure: 114 -> assert [("racoons are biting trash pandas", "USA", "bar")] == table.search( 115 "bite", order="rowid" 116 ) 117 118 119 def test_optimize_fts(fresh_db): (Pdb) table.search("bite") [(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)] The problem here is that the table.search() method now behaves differently for FTS4 v.s. FTS5 tables.

With FTS4 you get back just the table columns.

With FTS5 you also get back the rowid as the first column and the rank score as the last column.

This is weird. It also makes me question whether having .search() return a list of tuples is the right API design.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/197/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
735532751 MDU6SXNzdWU3MzU1MzI3NTE= 192 sqlite-utils search command simonw 9599 closed 0   3.0 6079500 9 2020-11-03T18:07:59Z 2020-11-08T17:07:01Z 2020-11-08T17:07:01Z OWNER  

A command that knows how to run a search against a FTS enabled table and return results ranked by relevance.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/192/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
738128913 MDU6SXNzdWU3MzgxMjg5MTM= 201 .search(columns=) and sqlite-utils search -c ... bug simonw 9599 closed 0   3.0 6079500 1 2020-11-07T01:27:26Z 2020-11-08T16:54:15Z 2020-11-08T16:54:15Z OWNER  

Both table.search(columns=) and the sqlite-utils search -c option do not work as expected - they always return both the rowid and the rank columns even if those have not been requested.

This should be fixed before the 3.0 non-alpha release.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/201/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
738115165 MDU6SXNzdWU3MzgxMTUxNjU= 200 sqlite-utils rows -c option simonw 9599 closed 0   3.0 6079500 1 2020-11-07T00:22:12Z 2020-11-07T00:28:48Z 2020-11-07T00:28:47Z OWNER  

To let you specify the exact columns you want. Based on the -c option to sqlite-utils search in #192.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/200/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
735648209 MDU6SXNzdWU3MzU2NDgyMDk= 193 --tsv output format option simonw 9599 closed 0   3.0 6079500 0 2020-11-03T21:31:18Z 2020-11-07T00:09:52Z 2020-11-07T00:09:52Z OWNER  

We already support --csv for output, and the insert command accepts --tsv. The output format options should accept --tsv too.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/193/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
577302229 MDU6SXNzdWU1NzczMDIyMjk= 91 Enable ordering FTS results by rank gfrmin 416374 closed 0   3.0 6079500 1 2020-03-07T08:43:51Z 2020-11-06T23:53:26Z 2020-11-06T23:53:25Z NONE  

According to https://www.sqlite.org/fts5.html (not sure about FTS4) results can be sorted by relevance. At the moment results are returned by default by rowid. Perhaps a flag can be added to the search method?

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/91/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);