issues

Table actions

16 rows where milestone = 7558727 and repo = 140912432 sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: title, user, comments, closed_at, author_association, created_at (date), updated_at (date), closed_at (date)

type 2

state 1

repo 1

  • sqlite-utils · 16 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1067771698 I_kwDOCGYnMM4_pOcy 348 Command for creating an empty database simonw 9599 closed 0   3.21 7558727 6 2021-11-30T23:24:27Z 2022-01-13T07:06:59Z 2022-01-09T20:33:20Z OWNER  

I sometimes find the need to create an empty SQLite database file - for example if I want to enable WAL on it before using it with another script. I currently do that like this:

sqlite3 my.db vacuum
sqlite-utils enable-wal my.db

It would be nice if sqlite-utils had a convenience command for doing this.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/348/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1096558279 I_kwDOCGYnMM5BXCbH 365 create-index should run analyze after creating index fgregg 536941 closed 0   3.21 7558727 16 2022-01-07T18:21:25Z 2022-01-11T02:43:34Z 2022-01-11T01:36:48Z CONTRIBUTOR  

sqlite's query planner depends upon analyze to make good use of indices. It would be nice if analyze was run as part of the create-index command.

If data is inserted later, things can get out date, but it would still probably be a net win.

sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1098574572 I_kwDOCGYnMM5Beurs 380 Release notes for 3.21 simonw 9599 closed 0   3.21 7558727 1 2022-01-11T02:12:30Z 2022-01-11T02:34:26Z 2022-01-11T02:34:26Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/380/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097251014 I_kwDOCGYnMM5BZrjG 375 `sqlite-utils bulk` command simonw 9599 closed 0   3.21 7558727 3 2022-01-09T17:12:38Z 2022-01-11T02:12:58Z 2022-01-11T02:10:55Z OWNER  

The .executemany() method is a very efficient way to execute the same SQL query against a huge list of parameters.

sqlite-utils insert supports a bunch of ways of loading a list of dictionaries - from CSV, TSV, JSON, newline JSON and more thanks to: - #361

What if you could load a list of dictionaries and provide a SQL query with :named parameters that correspond to keys in those dictionaries instead?

This would need to be a new command - I thought about adding a --sql option to insert but that doesn't make sense as that command already requires a table name.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/375/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097477582 PR_kwDOCGYnMM4wtl17 377 `sqlite-utils bulk` command simonw 9599 closed 0   3.21 7558727 3 2022-01-10T05:34:24Z 2022-01-11T02:10:57Z 2022-01-11T02:10:54Z OWNER simonw/sqlite-utils/pulls/377

Refs #375

Still needs:

  • [x] Refactor @insert_upsert_options so that it doesn't duplicate @import_options
  • [x] Tests
  • [x] Documentation
  • [x] Try it against a really big file
 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/377/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1098544628 I_kwDOCGYnMM5BenX0 379 CLI options for running ANALYZE simonw 9599 closed 0   3.21 7558727 0 2022-01-11T01:09:16Z 2022-01-11T01:38:01Z 2022-01-11T01:36:48Z OWNER  

The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865

  • [x] sqlite-utils analyze command
  • [x] sqlite-utils create-index --analyze option (see #365)
  • [x] sqlite-utils insert --analyze option
  • [x] sqlite-utils upsert --analyze option

In #378 I also added .delete_where(..., analyze=True) but there isn't currently a sqlite-utils delete-where CLI command - deletions via CLI are expected to be handled using SQL queries.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/379/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1096563265 I_kwDOCGYnMM5BXDpB 366 Python library methods for calling ANALYZE simonw 9599 closed 0   3.21 7558727 10 2022-01-07T18:28:01Z 2022-01-11T01:09:33Z 2022-01-11T01:09:33Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1098309897 I_kwDOCGYnMM5BduEJ 378 analyze=True parameter for some methods simonw 9599 closed 0   3.21 7558727 0 2022-01-10T19:54:52Z 2022-01-11T01:08:11Z 2022-01-11T01:08:09Z OWNER  

This would cause ANALYZE to be run against the relevant table at the end of executing the method.

Having browsed the API reference I think the methods that would benefit from an analyze=True parameter are:

  • [x] table.create_index
  • [x] table.insert_all
  • [x] table.upsert_all
  • [x] table.delete_where

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/378/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097041471 PR_kwDOCGYnMM4wsVM6 367 Initial prototype of .analyze() methods simonw 9599 closed 0   3.21 7558727 2 2022-01-08T21:35:12Z 2022-01-10T19:31:08Z 2022-01-10T19:31:08Z OWNER simonw/sqlite-utils/pulls/367

Refs #366

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/367/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 1  
1097436959 I_kwDOCGYnMM5BaY8f 376 `--nl` mode should ignore blank lines simonw 9599 closed 0   3.21 7558727 0 2022-01-10T04:10:54Z 2022-01-10T19:27:41Z 2022-01-10T04:12:46Z OWNER  

Spotted this while manually testing #364 - there's no reason --nl should crash if you feed it an empty line in between JSON objects.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/376/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097129710 I_kwDOCGYnMM5BZN7u 372 Idea: `suffix` and `stem` file columns simonw 9599 closed 0   3.21 7558727 1 2022-01-09T07:48:53Z 2022-01-10T19:27:34Z 2022-01-09T20:17:00Z OWNER  

For https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files

Given a file called dogs.jpg stem would be dogs and ext would be jpg.

Need to decide what happens for dogs.and.cats.jpg.gz.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/372/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097128334 I_kwDOCGYnMM5BZNmO 371 Support mutating row in `--convert` without returning it simonw 9599 closed 0   3.21 7558727 6 2022-01-09T07:38:44Z 2022-01-10T19:27:30Z 2022-01-09T20:06:15Z OWNER  

Currently you have to do this: $ sqlite-utils insert dogs.db dogs dogs.json --convert ' row["is_good"] = 1 return row' Would be neat if this worked too: $ sqlite-utils insert dogs.db dogs dogs.json \ --convert 'row["is_good"] = 1'

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/371/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097135860 I_kwDOCGYnMM5BZPb0 374 `--fmt` should imply `-t` simonw 9599 closed 0   3.21 7558727 4 2022-01-09T08:23:07Z 2022-01-10T19:27:26Z 2022-01-09T18:07:59Z OWNER  

Not sure why I didn't implement this.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/374/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097135732 I_kwDOCGYnMM5BZPZ0 373 List `--fmt` options in the docs simonw 9599 closed 0   3.21 7558727 3 2022-01-09T08:22:11Z 2022-01-10T19:27:24Z 2022-01-09T17:49:00Z OWNER  

https://sqlite-utils.datasette.io/en/stable/cli.html#table-formatted-output currently cheats and tells the user to run --help - can fix this using cog.

sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/373/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097087280 I_kwDOCGYnMM5BZDkw 368 Offer `python -m sqlite_utils` as an alternative to `sqlite-utils` simonw 9599 closed 0   3.21 7558727 3 2022-01-09T02:29:30Z 2022-01-10T19:27:20Z 2022-01-09T02:40:50Z OWNER  

Add this to sqlite_utils/cli.py:

python if __name__ == "__main__": cli() Now the tool can be run using python -m sqlite_utils.cli --help

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/368/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1095570074 I_kwDOCGYnMM5BTRKa 364 `--batch-size 1` doesn't seem to commit for every item simonw 9599 closed 0   3.21 7558727 16 2022-01-06T18:18:50Z 2022-01-10T19:27:17Z 2022-01-10T05:36:19Z OWNER  

I'm trying this, but it doesn't seem to write anything to the database file until I hit CTRL+C:

heroku logs --app=simonwillisonblog --tail | grep 'measure#nginx.service' | \ sqlite-utils insert /tmp/herokutail.db log - --import re --convert "$(cat <<EOD r = re.compile(r'([^\s=]+)=(?:"(.*?)"|(\S+))') pairs = {} for key, value1, value2 in r.findall(line): pairs[key] = value1 or value2 return pairs EOD )" --lines --batch-size 1

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);