issues

Table actions

78 rows where comments = 0, repo = 140912432 and user = 9599 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: milestone, draft, created_at (date), updated_at (date), closed_at (date)

type 2

state 2

repo 1

  • sqlite-utils · 78 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1977155641 I_kwDOCGYnMM512QA5 601 Move plugin directory into documentation simonw 9599 open 0     0 2023-11-04T04:07:52Z 2023-11-04T04:07:52Z   OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/601/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1786243905 I_kwDOCGYnMM5qd-tB 564 Document that running `db.transform()` tidies up the schema indentation simonw 9599 closed 0     0 2023-07-03T13:59:28Z 2023-07-22T22:15:34Z 2023-07-22T22:15:34Z OWNER  

... and it turns out running .transform() with no arguments still fixes the format of the schema!

```pycon

db["log"].add_column("foo", str)

<Table log (id, name2, age, weight, foo)> >>> db["log"].add_column("bar", str) <Table log (id, name2, age, weight, foo, bar)> >>> db["log"].add_column("baz", str) <Table log (id, name2, age, weight, foo, bar, baz)> >>> print(db["log"].schema) CREATE TABLE "log" ( [id] INTEGER PRIMARY KEY, [name2] TEXT, [age] INTEGER, [weight] FLOAT , [foo] TEXT, [bar] TEXT, [baz] TEXT) >>> db["log"].transform() <Table log (id, name2, age, weight, foo, bar, baz)> >>> print(db["log"].schema) CREATE TABLE "log" ( [id] INTEGER PRIMARY KEY, [name2] TEXT, [age] INTEGER, [weight] FLOAT, [foo] TEXT, [bar] TEXT, [baz] TEXT ) ``` _Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1618347727_
 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/564/reactions",
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 1
}
   completed
1816857105 I_kwDOCGYnMM5sSwoR 570 `sqlite-utils install -e` option simonw 9599 closed 0     0 2023-07-22T18:32:23Z 2023-07-22T18:55:59Z 2023-07-22T18:32:56Z OWNER  

As seen in LLM.

Needed while working on: - #567

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/570/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1773450152 I_kwDOCGYnMM5ptLOo 559 sqlean support simonw 9599 closed 0     0 2023-06-25T19:27:26Z 2023-06-25T23:25:53Z 2023-06-25T23:25:53Z OWNER  

If sqlean is available, use that.

Refs: - https://github.com/nalgeon/sqlean.py/issues/1#issuecomment-1605707788

This will provide a good workaround for: - #235

sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/559/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1701018909 I_kwDOCGYnMM5lY30d 543 Tests broken on Windows due to new convert() lambda names simonw 9599 closed 0     0 2023-05-08T22:11:29Z 2023-05-08T22:19:04Z 2023-05-08T22:19:04Z OWNER  

https://github.com/simonw/sqlite-utils/actions/runs/4920084038/jobs/8788501314 python sql = 'update [example] set [dt] = lambda_-9223371942137158589([dt]);' From: - #526

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/543/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1487764628 I_kwDOCGYnMM5YrXyU 518 flake8 ValueError: Error code '#' supplied to 'extend-ignore' option... simonw 9599 closed 0     0 2022-12-10T01:30:24Z 2022-12-10T01:36:46Z 2022-12-10T01:36:46Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/518/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1359604075 I_kwDOCGYnMM5RCelr 481 Idea: `sqlite-utils create-table tablename --sql "select ..."` simonw 9599 open 0     0 2022-09-02T01:41:24Z 2022-09-02T01:42:08Z   OWNER  

Could offer syntactic sugar for:

sql create table foo as select * from bar

sqlite-utils create-table data.db foo --sql "select * from bar" https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/481/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1342374388 PR_kwDOCGYnMM49Wv9T 466 Use Read the Docs action v1 (#463) simonw 9599 closed 0     0 2022-08-17T23:11:50Z 2022-08-17T23:11:54Z 2022-08-17T23:11:54Z OWNER simonw/sqlite-utils/pulls/466

Read the Docs repository was renamed from readthedocs/readthedocs-preview to readthedocs/actions/. Now, the preview action is under readthedocs/actions/preview and is tagged as v1

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/466/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1243715381 I_kwDOCGYnMM5KIZc1 436 Add "copy to clipboard" button to code examples in documentation simonw 9599 closed 0     0 2022-05-20T21:53:23Z 2022-05-20T21:57:53Z 2022-05-20T21:57:53Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/436/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1123851690 I_kwDOCGYnMM5C_J2q 396 mypy failure, sqlite_utils/utils.py:56 simonw 9599 closed 0     0 2022-02-04T06:08:09Z 2022-02-04T06:10:33Z 2022-02-04T06:10:33Z OWNER  

https://github.com/simonw/sqlite-utils/runs/5062725880?check_suite_focus=true

sqlite_utils/utils.py:56: error: Incompatible return value type (got "None", expected "str")

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/396/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1099897648 I_kwDOCGYnMM5Bjxsw 384 Add examples to every `--help` simonw 9599 closed 0     0 2022-01-12T05:31:25Z 2022-01-26T03:15:02Z 2022-01-26T03:15:02Z OWNER  

Everything on https://sqlite-utils.datasette.io/en/stable/cli-reference.html would benefit from an example.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/384/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1107557831 I_kwDOCGYnMM5CA_3H 386 Better "contributing" documentation simonw 9599 closed 0     0 2022-01-19T02:11:48Z 2022-01-19T02:15:21Z 2022-01-19T02:15:21Z OWNER  

This page jumps straight into running the tests: https://sqlite-utils.datasette.io/en/latest/contributing.html

It should add a little more about expected collaboration styles - opening an issue before filing a pull request - and probably link to https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/386/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1098544628 I_kwDOCGYnMM5BenX0 379 CLI options for running ANALYZE simonw 9599 closed 0   3.21 7558727 0 2022-01-11T01:09:16Z 2022-01-11T01:38:01Z 2022-01-11T01:36:48Z OWNER  

The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865

  • [x] sqlite-utils analyze command
  • [x] sqlite-utils create-index --analyze option (see #365)
  • [x] sqlite-utils insert --analyze option
  • [x] sqlite-utils upsert --analyze option

In #378 I also added .delete_where(..., analyze=True) but there isn't currently a sqlite-utils delete-where CLI command - deletions via CLI are expected to be handled using SQL queries.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/379/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1098309897 I_kwDOCGYnMM5BduEJ 378 analyze=True parameter for some methods simonw 9599 closed 0   3.21 7558727 0 2022-01-10T19:54:52Z 2022-01-11T01:08:11Z 2022-01-11T01:08:09Z OWNER  

This would cause ANALYZE to be run against the relevant table at the end of executing the method.

Having browsed the API reference I think the methods that would benefit from an analyze=True parameter are:

  • [x] table.create_index
  • [x] table.insert_all
  • [x] table.upsert_all
  • [x] table.delete_where

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/378/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097436959 I_kwDOCGYnMM5BaY8f 376 `--nl` mode should ignore blank lines simonw 9599 closed 0   3.21 7558727 0 2022-01-10T04:10:54Z 2022-01-10T19:27:41Z 2022-01-10T04:12:46Z OWNER  

Spotted this while manually testing #364 - there's no reason --nl should crash if you feed it an empty line in between JSON objects.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/376/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1094974713 I_kwDOCGYnMM5BQ_z5 362 upsert --detect-types is broken simonw 9599 closed 0     0 2022-01-06T05:12:10Z 2022-01-06T06:54:45Z 2022-01-06T06:28:34Z OWNER  

Noticed this thanks to syntax highlighting in VS Code showing an unused variable - need to fix it and add a test.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/362/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1090798237 I_kwDOCGYnMM5BBEKd 359 Use RETURNING if available to populate last_pk simonw 9599 open 0     0 2021-12-29T23:43:23Z 2021-12-29T23:43:23Z   OWNER  

Inspired by this: https://news.ycombinator.com/item?id=29729283

Because SQLite is effectively serializing all the writes for us, we have zero locking in our code. We used to have to lock when inserting new items (to get the LastInsertRowId), but the newer version of SQLite supports the RETURNING keyword, so we don't even have to lock on inserts now.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/359/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1053136495 I_kwDOCGYnMM4-xZZv 341 `hash_id: Optional[Any]` should be `hash_id: Optional[str]` simonw 9599 closed 0     0 2021-11-15T02:12:39Z 2021-11-15T02:19:31Z 2021-11-15T02:19:31Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/341/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1053087862 I_kwDOCGYnMM4-xNh2 338 dict, list, tuple should all map to TEXT simonw 9599 closed 0     0 2021-11-15T00:28:01Z 2021-11-15T00:36:03Z 2021-11-15T00:36:03Z OWNER  

This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947

So the COLUMN_TYPE_MAPPING should include those too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/322#issuecomment-968401459

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/338/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
931752773 MDU6SXNzdWU5MzE3NTI3NzM= 294 Add a `sqlite-utils memory` example to the README simonw 9599 closed 0     0 2021-06-28T16:35:59Z 2021-08-18T21:40:03Z 2021-08-18T21:40:03Z OWNER  
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/294/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
965166058 MDU6SXNzdWU5NjUxNjYwNTg= 313 `.add_foreign_keys()` doesn't reject being called with a View simonw 9599 closed 0     0 2021-08-10T17:22:17Z 2021-08-10T17:25:34Z 2021-08-10T17:25:34Z OWNER  

Spotted this bug using mypy while working on #311 / #312!

% mypy sqlite_utils sqlite_utils/db.py:725: error: Item "View" of "Union[Table, View]" has no attribute "foreign_keys" Found 1 error in 1 file (checked 5 source files) Refers to this code: https://github.com/simonw/sqlite-utils/blob/c11ff89894727270d4a9eb554d3a006f5b0d8d9d/sqlite_utils/db.py#L710-L720

It's a bug! We run some checks earlier but none of them ensure that it's a view:

https://github.com/simonw/sqlite-utils/blob/c11ff89894727270d4a9eb554d3a006f5b0d8d9d/sqlite_utils/db.py#L697-L709

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/313/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
959305209 MDU6SXNzdWU5NTkzMDUyMDk= 307 codespell to spell check documentation simonw 9599 closed 0     0 2021-08-03T16:48:19Z 2021-08-03T16:48:53Z 2021-08-03T16:48:53Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/307/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
957383814 MDU6SXNzdWU5NTczODM4MTQ= 301 insert-files should get a --silent option simonw 9599 closed 0     0 2021-08-01T04:11:03Z 2021-08-02T19:12:21Z 2021-08-02T19:12:21Z OWNER  

The new sqlite-utils convert command I'm adding in #251 will have a --silent option for turning off the progress bars. The only other command that has progress bars right now is insert-files so it should get this option too, for consistency.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/301/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
927789811 MDU6SXNzdWU5Mjc3ODk4MTE= 292 Add contributing documentation simonw 9599 closed 0     0 2021-06-23T02:13:05Z 2021-06-25T17:53:51Z 2021-06-25T17:53:51Z OWNER  

Like https://docs.datasette.io/en/latest/contributing.html (but simpler) - should cover how to run black and flake8 and mypy and how to run the tests.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/292/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
925544070 MDU6SXNzdWU5MjU1NDQwNzA= 287 Update rowid examples in the docs simonw 9599 closed 0     0 2021-06-20T08:03:00Z 2021-06-20T18:26:21Z 2021-06-20T18:26:21Z OWNER  

Changed in #284 - a couple of examples need updating on https://github.com/simonw/sqlite-utils/blob/3.10/docs/cli.rst.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/287/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
925545468 MDU6SXNzdWU5MjU1NDU0Njg= 288 sqlite-utils memory blah.json --schema simonw 9599 closed 0     0 2021-06-20T08:10:40Z 2021-06-20T18:26:21Z 2021-06-20T18:26:21Z OWNER  

Like --dump but only outputs the schema - useful for understanding what you are about to run queries against.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/288/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
924991194 MDU6SXNzdWU5MjQ5OTExOTQ= 280 Add --encoding option to sqlite-utils memory simonw 9599 closed 0     0 2021-06-18T15:03:32Z 2021-06-18T15:29:46Z 2021-06-18T15:29:46Z OWNER  

Follow-on from #272 - this will work like --encoding on sqlite-utils insert and will affect all CSV files processed by sqlite-utils memory.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/280/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
922832113 MDU6SXNzdWU5MjI4MzIxMTM= 274 sqlite-utils dump my.db command simonw 9599 closed 0     0 2021-06-16T16:30:14Z 2021-06-16T23:51:54Z 2021-06-16T23:51:54Z OWNER  

Inspired by the --dump mechanism I added to sqlite-utils memory here: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937

Can use .iterdump() to implement this: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump

Maybe instead (or as-well-as) offer --dump which dumps out the SQL from that.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/274/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
906355849 MDExOlB1bGxSZXF1ZXN0NjU3MzczNzI2 262 Ability to add descending order indexes simonw 9599 closed 0     0 2021-05-29T04:51:04Z 2021-05-29T05:01:42Z 2021-05-29T05:01:39Z OWNER simonw/sqlite-utils/pulls/262

Refs #260

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/262/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
858501079 MDU6SXNzdWU4NTg1MDEwNzk= 255 transform --help should tell you the available types simonw 9599 closed 0     0 2021-04-15T05:24:48Z 2021-05-29T03:55:52Z 2021-05-29T03:55:52Z OWNER  

``` Usage: sqlite-utils transform [OPTIONS] PATH TABLE

Transform a table beyond the capabilities of ALTER TABLE

Options: --type <TEXT TEXT>... Change column type to X ``` This should specify that the possible types are 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/255/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
816601354 MDExOlB1bGxSZXF1ZXN0NTgwMjM1NDI3 241 Extract expand - work in progress simonw 9599 open 0     0 2021-02-25T16:36:38Z 2021-02-25T16:36:38Z   OWNER simonw/sqlite-utils/pulls/241

Refs #239. Still needs documentation and CLI implementation.

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/241/reactions",
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 1  
808036774 MDU6SXNzdWU4MDgwMzY3NzQ= 232 Run tests against Windows in GitHub Actions simonw 9599 closed 0     0 2021-02-14T20:09:45Z 2021-02-14T20:39:55Z 2021-02-14T20:39:55Z OWNER  

I'm going to try and get the test suite to run in Windows on GitHub Actions.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/225#issuecomment-778834504

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/232/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
808037010 MDExOlB1bGxSZXF1ZXN0NTczMTQ3MTY4 233 Run tests against Ubuntu, macOS and Windows simonw 9599 closed 0     0 2021-02-14T20:11:02Z 2021-02-14T20:39:54Z 2021-02-14T20:39:54Z OWNER simonw/sqlite-utils/pulls/233

Refs #232

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/233/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
802583450 MDU6SXNzdWU4MDI1ODM0NTA= 226 3.4 release is broken - includes a rogue line simonw 9599 closed 0     0 2021-02-06T02:08:01Z 2021-02-06T02:10:26Z 2021-02-06T02:10:26Z OWNER  

I started seeing weird errors, caused by this line: https://github.com/simonw/sqlite-utils/blob/f8010ca78fed8c5fca6cde19658ec09fdd468420/sqlite_utils/cli.py#L1-L3

That was added by accident in 1b666f9315d4ea6bb332b2e75e48480c26100199

I'm surprised the tests didn't catch this!

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/226/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
787900412 MDU6SXNzdWU3ODc5MDA0MTI= 222 .m2m() should accept alter=True parameter simonw 9599 closed 0     0 2021-01-18T04:15:43Z 2021-01-18T04:26:10Z 2021-01-18T04:26:10Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/222/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
783910901 MDU6SXNzdWU3ODM5MTA5MDE= 221 .add_missing_columns() does not take case insensitivity into account simonw 9599 closed 0     0 2021-01-12T05:01:00Z 2021-01-12T23:17:33Z 2021-01-12T23:17:33Z OWNER  

SQLite columns are case insensitive - but the .add_missing_columns() method doesn't know that. This means that it can crash if it identifies a column that is a case-insensitive duplicate of an existing column. https://github.com/simonw/sqlite-utils/blob/4cc82fd0bccc9d2eeb3510beb4e691d7da099f84/sqlite_utils/db.py#L1974-L1980

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/221/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
777530107 MDU6SXNzdWU3Nzc1MzAxMDc= 214 sqlite-utils enable-counts command simonw 9599 closed 0     0 2021-01-02T21:45:48Z 2021-01-03T04:26:44Z 2021-01-03T04:26:44Z OWNER  

The CLI version of #212 and #213.

# Enable counts for all tables:
sqlite-utils enable-counts data.db

# Enable counts for specific tables:
sqlite-utils enable-counts data.db table1 table2
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/214/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
777386465 MDU6SXNzdWU3NzczODY0NjU= 211 table.triggers_dict introspection property simonw 9599 closed 0     0 2021-01-02T02:04:00Z 2021-01-02T02:10:10Z 2021-01-02T02:10:10Z OWNER  

table.triggers currently returns a list of Trigger values. A table.triggers_dict property could behave like columns_dict, returning a dictionary mapping trigger names to their SQL definitions for that table.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/211/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
735648209 MDU6SXNzdWU3MzU2NDgyMDk= 193 --tsv output format option simonw 9599 closed 0   3.0 6079500 0 2020-11-03T21:31:18Z 2020-11-07T00:09:52Z 2020-11-07T00:09:52Z OWNER  

We already support --csv for output, and the insert command accepts --tsv. The output format options should accept --tsv too.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/193/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
730693696 MDExOlB1bGxSZXF1ZXN0NTEwOTU2MTM0 190 Progress bar for sqlite-utils insert command simonw 9599 closed 0     0 2020-10-27T18:08:53Z 2020-10-27T18:16:03Z 2020-10-27T18:16:03Z OWNER simonw/sqlite-utils/pulls/190

Refs #173

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/190/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
718952107 MDU6SXNzdWU3MTg5NTIxMDc= 185 Use db[table] consistently in documentation simonw 9599 closed 0     0 2020-10-11T23:39:12Z 2020-10-12T00:13:41Z 2020-10-12T00:13:41Z OWNER  

The Python docs have a bunch of examples like this: https://sqlite-utils.readthedocs.io/en/stable/python-api.html python dogs.enable_fts(["name", "twitter"], create_triggers=True) This would be easier for people to understand if it looked like this instead: python db["dogs"].enable_fts(["name", "twitter"], create_triggers=True)

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/185/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
716955793 MDExOlB1bGxSZXF1ZXN0NDk5NjAzMzU5 184 Test against Python 3.9 simonw 9599 closed 0     0 2020-10-08T01:37:05Z 2020-10-08T01:44:06Z 2020-10-08T01:44:06Z OWNER simonw/sqlite-utils/pulls/184
sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/184/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
581795570 MDU6SXNzdWU1ODE3OTU1NzA= 93 Support more string values for types in .add_column() simonw 9599 open 0     0 2020-03-15T19:32:49Z 2020-09-24T20:36:46Z   OWNER  

https://sqlite-utils.readthedocs.io/en/2.4.2/python-api.html#adding-columns says:

SQLite types you can specify are "TEXT", "INTEGER", "FLOAT" or "BLOB".

As discovered in #92 this isn't the right list of values. I should expand this to match https://www.sqlite.org/datatype3.html

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/93/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
688352145 MDU6SXNzdWU2ODgzNTIxNDU= 141 insert-files support for compressed values simonw 9599 open 0     0 2020-08-28T20:59:46Z 2020-09-24T20:36:08Z   OWNER  

The sqlar format supports this, it would be useful if insert-files could support this too.

https://www.sqlite.org/sqlar.html

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/141/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
706091046 MDU6SXNzdWU3MDYwOTEwNDY= 165 Make .transform() a keyword arguments only function simonw 9599 closed 0   2.20 5897911 0 2020-09-22T05:37:29Z 2020-09-24T20:35:47Z 2020-09-22T06:39:12Z OWNER  

And rename the first argument from columns= to types=

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/165/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
706757891 MDU6SXNzdWU3MDY3NTc4OTE= 169 Progress bar for "sqlite-utils extract" simonw 9599 closed 0   2.20 5897911 0 2020-09-22T23:40:21Z 2020-09-24T20:34:50Z 2020-09-23T00:02:40Z OWNER  

Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command.

The operations are full table scans so calculating progress shouldn't be too difficult. Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246831

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/169/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
557830332 MDExOlB1bGxSZXF1ZXN0MzY5MzQ4MDg0 78 New conversions= feature, refs #77 simonw 9599 closed 0     0 2020-01-31T00:02:33Z 2020-09-22T07:48:29Z 2020-01-31T00:24:31Z OWNER simonw/sqlite-utils/pulls/78
sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/78/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
706092617 MDExOlB1bGxSZXF1ZXN0NDkwNzAzMTcz 166 Keyword only arguments for transform() simonw 9599 closed 0     0 2020-09-22T05:41:44Z 2020-09-22T06:39:11Z 2020-09-22T06:39:11Z OWNER simonw/sqlite-utils/pulls/166

Refs #165

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/166/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
695359607 MDU6SXNzdWU2OTUzNTk2MDc= 150 Feature for tracing SQL queries simonw 9599 closed 0     0 2020-09-07T19:43:08Z 2020-09-07T21:57:01Z 2020-09-07T21:57:01Z OWNER  

Debugging sqlite-utils when something weird happens (e.g. #149) can be a bit tricky since it runs a bunch of different SQL statements behind the scenes.

An optional "tracing" mechanism for seeing what SQL is being executed would be useful.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/150/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
695360889 MDExOlB1bGxSZXF1ZXN0NDgxNjE2NzA0 151 Tracer mechanism for seeing underlying SQL simonw 9599 closed 0     0 2020-09-07T19:46:43Z 2020-09-07T21:57:00Z 2020-09-07T21:57:00Z OWNER simonw/sqlite-utils/pulls/151

Refs #150. Needs tests and documentation, including for the new db.execute() and db.executescript() methods.

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/151/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
665701216 MDU6SXNzdWU2NjU3MDEyMTY= 123 --raw option for outputting binary content simonw 9599 closed 0     0 2020-07-26T03:35:39Z 2020-07-26T16:44:11Z 2020-07-26T16:44:11Z OWNER  

Related to the insert-files work in #122 - it should be easy to get binary data back out of the database again.

One way to do that could be:

sqlite-utils files.db "select content from files where key = 'foo.jpg'" --raw

The --raw option would cause just the contents of the first column to be output directly to stdout.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/123/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
602569315 MDU6SXNzdWU2MDI1NjkzMTU= 102 Can't store an array or dictionary containing a bytes value simonw 9599 closed 0     0 2020-04-18T22:49:21Z 2020-05-01T20:45:45Z 2020-05-01T20:45:45Z OWNER  

``` In [1]: import sqlite_utils

In [2]: db = sqlite_utils.Database(memory=True)

In [3]: db["t"].insert({"id": 1, "data": {"foo": b"bytes"}})

TypeError Traceback (most recent call last) <ipython-input-3-a8ab1f72c72c> in <module> ----> 1 db["t"].insert({"id": 1, "data": {"foo": b"bytes"}})

~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert(self, record, pk, foreign_keys, column_order, not_null, defaults, hash_id, alter, ignore, replace, extracts, conversions, columns) 950 extracts=extracts, 951 conversions=conversions, --> 952 columns=columns, 953 ) 954

~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, conversions, columns, upsert) 1052 for key in all_columns: 1053 value = jsonify_if_needed( -> 1054 record.get(key, None if key != hash_id else _hash(record)) 1055 ) 1056 if key in extracts:

~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in jsonify_if_needed(value) 1318 def jsonify_if_needed(value): 1319 if isinstance(value, (dict, list, tuple)): -> 1320 return json.dumps(value) 1321 elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)): 1322 return value.isoformat()

/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/init.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw) 229 cls is None and indent is None and separators is None and 230 default is None and not sort_keys and not kw): --> 231 return _default_encoder.encode(obj) 232 if cls is None: 233 cls = JSONEncoder

/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in encode(self, o) 197 # exceptions aren't as detailed. The list call should be roughly 198 # equivalent to the PySequence_Fast that ''.join() would do. --> 199 chunks = self.iterencode(o, _one_shot=True) 200 if not isinstance(chunks, (list, tuple)): 201 chunks = list(chunks)

/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in iterencode(self, o, _one_shot) 255 self.key_separator, self.item_separator, self.sort_keys, 256 self.skipkeys, _one_shot) --> 257 return _iterencode(o, 0) 258 259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in default(self, o) 177 178 """ --> 179 raise TypeError(f'Object of type {o.class.name} ' 180 f'is not JSON serializable') 181

TypeError: Object of type bytes is not JSON serializable ```

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/102/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
610853393 MDU6SXNzdWU2MTA4NTMzOTM= 104 --schema option to "sqlite-utils tables" simonw 9599 closed 0     0 2020-05-01T16:55:49Z 2020-05-01T17:12:37Z 2020-05-01T17:12:37Z OWNER  

Adds output showing the table schema.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/104/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
601392318 MDU6SXNzdWU2MDEzOTIzMTg= 101 README should include an example of CLI data insertion simonw 9599 closed 0     0 2020-04-16T19:45:37Z 2020-04-17T23:59:49Z 2020-04-17T23:59:49Z OWNER  

Maybe using curl from the GitHub API.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/101/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
586486367 MDU6SXNzdWU1ODY0ODYzNjc= 95 Columns with only null values are no longer created in the database simonw 9599 closed 0     0 2020-03-23T20:07:42Z 2020-03-23T20:31:15Z 2020-03-23T20:31:15Z OWNER  

Bug introduced in #94, and released in 2.4.3.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/95/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
586477757 MDU6SXNzdWU1ODY0Nzc3NTc= 94 If column data is a mixture of integers and nulls, detected type should be INTEGER simonw 9599 closed 0     0 2020-03-23T19:51:46Z 2020-03-23T19:57:10Z 2020-03-23T19:57:10Z OWNER  

It looks like detected type for that case is TEXT at the moment.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/94/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
573740712 MDU6SXNzdWU1NzM3NDA3MTI= 90 Cannot .enable_fts() for columns with spaces in their names simonw 9599 closed 0     0 2020-03-02T06:06:03Z 2020-03-02T06:10:49Z 2020-03-02T06:10:49Z OWNER  

``` import sqlite_utils db = sqlite_utils.Database(memory=True)
db["test"].insert({"space in name": "hello"})
db["test"].enable_fts(["space in name"])

OperationalError Traceback (most recent call last) <ipython-input-8-ce4b87dd1c7a> in <module> ----> 1 db['test'].enable_fts(["space in name"])

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in enable_fts(self, columns, fts_version, create_triggers) 755 ) 756 self.db.conn.executescript(sql) --> 757 self.populate_fts(columns) 758 759 if create_triggers:

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in populate_fts(self, columns) 787 table=self.name, columns=", ".join(columns) 788 ) --> 789 self.db.conn.executescript(sql) 790 return self 791

OperationalError: near "in": syntax error ```

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/90/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
561460274 MDU6SXNzdWU1NjE0NjAyNzQ= 84 .upsert() with hash_id throws error simonw 9599 closed 0     0 2020-02-07T07:08:19Z 2020-02-07T07:17:11Z 2020-02-07T07:17:11Z OWNER  

python db[table_name].upsert_all(rows, hash_id="pk") This throws an error: PrimaryKeyRequired('upsert() requires a pk')

The problem is, if you try this:

python db[table_name].upsert_all(rows, hash_id="pk", pk="pk") You get this error: AssertionError('Use either pk= or hash_id=')

hash_id= should imply that pk= that column.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/84/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
543738004 MDExOlB1bGxSZXF1ZXN0MzU3OTkyNTg4 72 Fixed implementation of upsert simonw 9599 closed 0     0 2019-12-30T05:08:05Z 2019-12-30T05:29:24Z 2019-12-30T05:29:24Z OWNER simonw/sqlite-utils/pulls/72

Refs #66

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/72/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
519039316 MDExOlB1bGxSZXF1ZXN0MzM3ODUzMzk0 65 Release 1.12.1 simonw 9599 closed 0     0 2019-11-07T04:51:29Z 2019-11-07T04:58:48Z 2019-11-07T04:58:47Z OWNER simonw/sqlite-utils/pulls/65
sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/65/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
519032008 MDExOlB1bGxSZXF1ZXN0MzM3ODQ3NTcz 64 test_insert_upsert_all_empty_list simonw 9599 closed 0     0 2019-11-07T04:24:45Z 2019-11-07T04:32:38Z 2019-11-07T04:32:38Z OWNER simonw/sqlite-utils/pulls/64
sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/64/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
488338965 MDU6SXNzdWU0ODgzMzg5NjU= 59 Ability to introspect triggers simonw 9599 closed 0     0 2019-09-02T23:47:16Z 2019-09-03T01:52:36Z 2019-09-03T00:09:42Z OWNER  

Now that we're creating triggers (thanks to @amjith in #57) it would be neat if we could introspect them too.

I'm thinking:

db.triggers - lists all triggers for the database db["tablename"].triggers - lists triggers for that table

The underlying query for this is select * from sqlite_master where type = 'trigger'

I'll return the trigger information in a new namedtuple, similar to how Indexes and ForeignKeys work.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/59/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
488341021 MDExOlB1bGxSZXF1ZXN0MzEzMzgzMzE3 60 db.triggers and table.triggers introspection simonw 9599 closed 0     0 2019-09-03T00:04:32Z 2019-09-03T00:09:42Z 2019-09-03T00:09:42Z OWNER simonw/sqlite-utils/pulls/60

Closes #59

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/60/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
476436920 MDExOlB1bGxSZXF1ZXN0MzAzOTkwNjgz 53 Work in progress: m2m() method for creating many-to-many records simonw 9599 closed 0     0 2019-08-03T10:03:56Z 2019-08-04T03:38:10Z 2019-08-04T03:37:33Z OWNER simonw/sqlite-utils/pulls/53
  • [x] table.insert({"name": "Barry"}).m2m("tags", lookup={"tag": "Coworker"})
  • [x] Explicit table name .m2m("humans", ..., m2m_table="relationships")
  • [x] Automatically use an existing m2m table if a single obvious candidate exists (a table with two foreign keys in the correct directions)
  • [x] Require the explicit m2m_table= argument if multiple candidates for the m2m table exist
  • [x] Documentation

Refs #23

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/53/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
471797101 MDExOlB1bGxSZXF1ZXN0MzAwMzc3NTk5 47 extracts= table parameter simonw 9599 closed 0     0 2019-07-23T16:30:29Z 2019-07-23T17:00:43Z 2019-07-23T17:00:43Z OWNER simonw/sqlite-utils/pulls/47

Still needs docs. Refs #46

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/47/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
471684708 MDExOlB1bGxSZXF1ZXN0MzAwMjg2NTM1 45 Implemented table.lookup(...), closes #44 simonw 9599 closed 0     0 2019-07-23T13:03:30Z 2019-07-23T13:07:00Z 2019-07-23T13:07:00Z OWNER simonw/sqlite-utils/pulls/45
sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/45/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
470131537 MDU6SXNzdWU0NzAxMzE1Mzc= 41 sqlite-utils insert --tsv option simonw 9599 closed 0     0 2019-07-19T04:27:21Z 2019-07-19T04:50:47Z 2019-07-19T04:50:47Z OWNER  

Right now we only support ingesting CSV, but sometimes interesting data is released as TSV.

https://www.washingtonpost.com/national/2019/07/18/how-download-use-dea-pain-pills-database/ for example.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/41/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
467864071 MDU6SXNzdWU0Njc4NjQwNzE= 39 table.get(...) method simonw 9599 closed 0     0 2019-07-14T17:20:51Z 2019-07-15T04:28:53Z 2019-07-15T04:28:53Z OWNER  

Utility method for fetching a record by its primary key.

Accepts a single value (for primary key / rowid tables) or a list/tuple of values (for compound primary keys, refs #36).

Raises a NotFoundError if the record cannot be found.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/39/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
462817589 MDU6SXNzdWU0NjI4MTc1ODk= 36 Support compound primary keys simonw 9599 closed 0     0 2019-07-01T17:00:07Z 2019-07-15T04:28:52Z 2019-07-15T04:28:52Z OWNER  

This should work: python table = db["dog_breeds"].insert({ "dog_id": 1, "breed_id": 2 }, pk=("dog_id", "breed_id")) Needed for m2m work in #23

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/36/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
462423972 MDExOlB1bGxSZXF1ZXN0MjkzMTE3MTgz 34 sqlite-utils index-foreign-keys / db.index_foreign_keys() simonw 9599 closed 0     0 2019-06-30T16:43:40Z 2019-06-30T23:50:55Z 2019-06-30T23:50:55Z OWNER simonw/sqlite-utils/pulls/34

Refs #33

  • [x] sqlite-utils index-foreign-keys command
  • [x] db.index_foreign_keys() method
  • [x] unit tests
  • [x] documentation
 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/34/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
461237618 MDU6SXNzdWU0NjEyMzc2MTg= 31 Mechanism for adding multiple foreign key constraints at once simonw 9599 closed 0     0 2019-06-27T00:04:30Z 2019-06-29T06:27:40Z 2019-06-29T06:27:40Z OWNER  

Needed by db-to-sqlite. It currently works by collecting all of the foreign key relationships it can find and then applying them at the end of the process.

The problem is, the add_foreign_key() method looks like this:

https://github.com/simonw/sqlite-utils/blob/86bd2bba689e25f09551d611ccfbee1e069e5b66/sqlite_utils/db.py#L498-L516

That means it's doing a full VACUUM for every single relationship it sets up - and if you have hundreds of foreign key relationships in your database this can take hours.

I think the right solution is to have a .add_foreign_keys(list_of_args) method which does the bulk operation and then a single VACUUM. .add_foreign_key(...) can then call the bulk action with a single list item.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/31/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
458941203 MDU6SXNzdWU0NTg5NDEyMDM= 29 Prevent accidental add-foreign-key with invalid column simonw 9599 closed 0     0 2019-06-20T23:57:24Z 2019-06-20T23:58:26Z 2019-06-20T23:58:26Z OWNER  

You can corrupt your database by running:

$ sqlite-utils add-foreign-key my.db table non_existent_column other_table other_column
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/29/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
432217625 MDU6SXNzdWU0MzIyMTc2MjU= 19 Incorrect help text for enable-fts command simonw 9599 closed 0   1.0 4348046 0 2019-04-11T19:46:44Z 2019-05-25T00:44:31Z 2019-05-25T00:44:31Z OWNER  

I clearly copied-and-pasted this from the tables command without updating it:

https://github.com/simonw/sqlite-utils/blob/0b1af42ead3b3902347951180b3364ce1942da6e/sqlite_utils/cli.py#L216-L222

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/19/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
413857257 MDU6SXNzdWU0MTM4NTcyNTc= 15 Ability to add columns to tables simonw 9599 closed 0     0 2019-02-24T19:20:51Z 2019-02-24T20:04:40Z 2019-02-24T20:04:40Z OWNER  

Makes sense to do this before foreign keys in #2

Python:

db["table"].add_column("new_column", int)

CLI:

$ sqlite-utils add-column table new_column INTEGER
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/15/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
413778585 MDExOlB1bGxSZXF1ZXN0MjU1NjU4MTEy 12 Support for numpy types, closes #11 simonw 9599 closed 0     0 2019-02-24T03:57:32Z 2019-02-24T04:02:20Z 2019-02-24T04:02:20Z OWNER simonw/sqlite-utils/pulls/12
sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/12/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
403396009 MDExOlB1bGxSZXF1ZXN0MjQ3ODYxNDE5 5 Run Travis tests against Python 3.8-dev simonw 9599 closed 0     0 2019-01-26T02:30:55Z 2019-01-26T02:37:54Z 2019-01-26T02:37:54Z OWNER simonw/sqlite-utils/pulls/5
sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/5/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
403028630 MDExOlB1bGxSZXF1ZXN0MjQ3NTc2OTQy 4 Fts5 simonw 9599 closed 0     0 2019-01-25T06:54:05Z 2019-01-25T06:54:33Z 2019-01-25T06:54:33Z OWNER simonw/sqlite-utils/pulls/4
sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/4/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
347058326 MDExOlB1bGxSZXF1ZXN0MjA1NzcwOTk2 1 Make .indexes compatible with older SQLite versions simonw 9599 closed 0     0 2018-08-02T15:17:05Z 2018-08-02T15:17:30Z 2018-08-02T15:17:30Z OWNER simonw/sqlite-utils/pulls/1

Older SQLite versions return a different set of columns from the PRAGMA we are using.

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);