issues

Table actions

73 rows where comments = 2, repo = 140912432 and user = 9599 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: milestone, draft, created_at (date), updated_at (date), closed_at (date)

type 2

state 2

repo 1

  • sqlite-utils · 73 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
2029161033 I_kwDOCGYnMM548opJ 606 str and int as aliases for text and integer simonw 9599 closed 0     2 2023-12-06T18:35:49Z 2023-12-06T19:44:04Z 2023-12-06T18:49:32Z OWNER  

I keep making this mistake: bash sqlite-utils add-column content.db assets _since int ``` Usage: sqlite-utils add-column [OPTIONS] PATH TABLE COL_NAME [[integer|float|b lob|text|INTEGER|FLOAT|BLOB|TEXT]] Try 'sqlite-utils add-column -h' for help.

Error: Invalid value for '[[integer|float|blob|text|INTEGER|FLOAT|BLOB|TEXT]]': 'int' is not one of 'integer', 'float', 'blob', 'text', 'INTEGER', 'FLOAT', 'BLOB', 'TEXT'. ```

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/606/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1891614971 I_kwDOCGYnMM5wv8D7 594 Represent compound foreign keys in table.foreign_keys output simonw 9599 open 0     2 2023-09-12T03:48:24Z 2023-09-12T03:51:13Z   OWNER  

Given this schema: sql CREATE TABLE departments ( campus_name TEXT NOT NULL, dept_code TEXT NOT NULL, dept_name TEXT, PRIMARY KEY (campus_name, dept_code) ); CREATE TABLE courses ( course_code TEXT PRIMARY KEY, course_name TEXT, campus_name TEXT NOT NULL, dept_code TEXT NOT NULL, FOREIGN KEY (campus_name, dept_code) REFERENCES departments(campus_name, dept_code) ); The output of db["courses"].foreign_keys right now is: [ForeignKey(table='courses', column='campus_name', other_table='departments', other_column='campus_name'), ForeignKey(table='courses', column='dept_code', other_table='departments', other_column='dept_code')] Which suggests two normal foreign keys, not one compound foreign key.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/594/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1816997390 I_kwDOCGYnMM5sTS4O 576 Backfill the release notes prior to 0.4 simonw 9599 closed 0     2 2023-07-23T05:41:42Z 2023-07-23T05:49:51Z 2023-07-23T05:48:21Z OWNER  

Currently the changelog starts at 0.4:

https://sqlite-utils.datasette.io/en/3.34/changelog.html#id115

I want the other releases - according to https://pypi.org/project/sqlite-utils/#history there are three missing:

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/576/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1816919568 I_kwDOCGYnMM5sS_4Q 575 Python API ability to opt-out of connection plugins simonw 9599 closed 0     2 2023-07-22T23:01:13Z 2023-07-22T23:17:22Z 2023-07-22T23:08:22Z OWNER  

Plugins affecting the CLI by default makes sense to me.

I'm less confident about them always affecting users of the Python API.

I'm going to have them apply by default, but I'm going to add a mechanism to opt-out on an individual database basis. Basically this:

```python from sqlite_utils import Database db = Database(memory=True, execute_plugins=False)

Anything using db from here on will not execute plugins

``` cc @asg017

Refs: - #567 - #574

sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/575/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1816877910 I_kwDOCGYnMM5sS1tW 572 Don't test Python 3.7 against textual simonw 9599 closed 0     2 2023-07-22T19:57:03Z 2023-07-22T22:16:50Z 2023-07-22T22:16:50Z OWNER  

Spotted this in the GitHub Actions logs:

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/572/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1718635018 PR_kwDOCGYnMM5Q9lY4 553 Reformatted CLI examples in docs simonw 9599 closed 0     2 2023-05-21T20:44:34Z 2023-05-21T20:57:27Z 2023-05-21T20:57:23Z OWNER simonw/sqlite-utils/pulls/553

Refs: - #551

:books: Documentation preview :books:: https://sqlite-utils--553.org.readthedocs.build/en/553/

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/553/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1718515590 I_kwDOCGYnMM5mbneG 544 New options for analyze-tables --common-limit --no-most and --no-least simonw 9599 closed 0     2 2023-05-21T14:03:19Z 2023-05-21T17:03:06Z 2023-05-21T16:19:31Z OWNER  

The "least common" section is frequently uninteresting, especially for huge tables with a large number of repeated-once values.

sqlite-utils analyze-tables content.db repos --common-limit 20 --no-least
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/544/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1718550688 PR_kwDOCGYnMM5Q9VH0 546 Analyze tables options: --common-limit, --no-most, --no-least simonw 9599 closed 0     2 2023-05-21T15:54:39Z 2023-05-21T16:19:30Z 2023-05-21T16:19:30Z OWNER simonw/sqlite-utils/pulls/546

Refs #544

  • [x] Documentation for CLI options
  • [x] Documentation for new Python API parameters: most_common: bool and least_common: bool
  • [x] Tests for CLI
  • [x] Tests for Python API
 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/546/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1620516340 I_kwDOCGYnMM5glx30 533 ReadTheDocs error: not all arguments converted during string formatting simonw 9599 closed 0     2 2023-03-12T21:21:05Z 2023-03-12T21:25:33Z 2023-03-12T21:25:33Z OWNER  

This came up as a failure running tests for: - #531

Traceback on https://readthedocs.org/projects/sqlite-utils/builds/19749348/

``` File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 889, in interpreted nodes, messages2 = role_fn(role, rawsource, text, lineno, self) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part TypeError: not all arguments converted during string formatting

Exception occurred: File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part TypeError: not all arguments converted during string formatting ```

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/533/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1423182778 I_kwDOCGYnMM5U1Au6 505 Release sqlite-utils 3.30 simonw 9599 closed 0     2 2022-10-25T22:20:05Z 2022-10-25T22:41:26Z 2022-10-25T22:41:16Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/505/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1386593843 I_kwDOCGYnMM5Spb4z 494 Document how to use Just simonw 9599 closed 0     2 2022-09-26T19:25:12Z 2022-09-26T19:32:36Z 2022-09-26T19:26:39Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/494/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1363765916 I_kwDOCGYnMM5RSWqc 483 `sqlite-utils install` command simonw 9599 closed 0     2 2022-09-06T20:13:55Z 2022-09-26T19:04:43Z 2022-09-26T18:57:15Z OWNER  

With the addition of --functions in: - #471

In addition to the existing convert command, there are now very good reasons to want to install additional packages into the same virtual environment as sqlite-utils itself, to allow them to be used with those features.

This isn't easy if you installed the tool with pipx or brew install sqlite-utils.

Datasette solved this problem with the datasette install command:

sqlite-utils could benefit from the same idea.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/483/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1353196970 I_kwDOCGYnMM5QqCWq 476 Release notes for 3.29 simonw 9599 closed 0   3.29 8355157 2 2022-08-27T23:21:21Z 2022-08-28T04:07:15Z 2022-08-28T04:07:03Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/476/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1352946135 I_kwDOCGYnMM5QpFHX 472 Reuse the locals/globals fix from --functions for other code accepting options simonw 9599 closed 0   3.29 8355157 2 2022-08-27T05:12:05Z 2022-08-27T05:20:12Z 2022-08-27T05:20:12Z OWNER  

I figured out a workaround for the ugly global x hack here: - https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120653

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/472/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1319881016 PR_kwDOCGYnMM48Mmde 457 Link to installation instructions simonw 9599 closed 0   3.29 8355157 2 2022-07-27T17:38:36Z 2022-08-27T03:55:52Z 2022-07-27T17:57:50Z OWNER simonw/sqlite-utils/pulls/457 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/457/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1342357149 PR_kwDOCGYnMM49Wsnq 465 beanbag-docutils>=2.0 simonw 9599 closed 0     2 2022-08-17T22:41:39Z 2022-08-17T23:38:07Z 2022-08-17T23:38:02Z OWNER simonw/sqlite-utils/pulls/465

Refs #464

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/465/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1326391841 PR_kwDOCGYnMM48iLGF 462 Discord badge simonw 9599 closed 0     2 2022-08-02T20:56:04Z 2022-08-02T21:15:57Z 2022-08-02T21:15:52Z OWNER simonw/sqlite-utils/pulls/462

Also testing fix for: - https://github.com/readthedocs/readthedocs-preview/issues/10

:books: Documentation preview :books:: https://sqlite-utils--462.org.readthedocs.build/en/462/

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/462/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1298531653 I_kwDOCGYnMM5NZgVF 451 Make sqlite_utils.utils.chunks a documented function simonw 9599 closed 0     2 2022-07-08T06:01:04Z 2022-07-15T22:09:34Z 2022-07-15T21:59:33Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/451/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1277328147 I_kwDOCGYnMM5MInsT 446 Use Just to automate running tests and linters locally simonw 9599 closed 0     2 2022-06-20T19:51:09Z 2022-06-21T19:28:35Z 2022-06-20T19:54:50Z OWNER  

I keep committing code that fails additional tests like mypy and flake8 and black. Automate those using Just.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/446/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1269998342 I_kwDOCGYnMM5LsqMG 443 Make `utils.rows_from_file()` a documented API simonw 9599 closed 0     2 2022-06-13T21:53:24Z 2022-06-20T19:49:37Z 2022-06-14T20:12:46Z OWNER  

rows_from_file() isn't part of the documented API but maybe it should be!

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/443/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1269886084 I_kwDOCGYnMM5LsOyE 442 `maximize_csv_field_size_limit()` utility function simonw 9599 closed 0     2 2022-06-13T19:54:54Z 2022-06-14T21:55:15Z 2022-06-14T21:31:49Z OWNER  

This code here runs only if cli.py is imported: https://github.com/simonw/sqlite-utils/blob/7ddf5300886a32d6daf60cf1d71efe492b65c87e/sqlite_utils/cli.py#L50-L59

I found myself needing the same fix in another library:

It should be a documented utility function.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/442/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1243704847 I_kwDOCGYnMM5KIW4P 435 Switch to Furo documentation theme simonw 9599 closed 0     2 2022-05-20T21:46:39Z 2022-05-20T21:56:10Z 2022-05-20T21:54:43Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/435/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1223177069 PR_kwDOCGYnMM43LrKB 429 Depend on click-default-group-wheel simonw 9599 closed 0     2 2022-05-02T18:03:10Z 2022-05-02T18:52:42Z 2022-05-02T18:05:00Z OWNER simonw/sqlite-utils/pulls/429

Trying to get this to work with Pyodide.

Refs: https://github.com/simonw/click-default-group-wheel/issues/3

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/429/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1128139375 I_kwDOCGYnMM5DPgpv 405 `Database(memory_name="name")` constructor argument simonw 9599 closed 0     2 2022-02-09T07:15:03Z 2022-02-16T01:23:16Z 2022-02-16T01:23:16Z OWNER  

SQLite in-memory databases can be named, in which case multiple connections can be opened to a shared in-memory database running within the same process.

Datasette supports this - SQLite could support it too.

https://docs.datasette.io/en/0.60.2/internals.html#database-ds-path-none-is-mutable-false-is-memory-false-memory-name-none

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/405/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1128120451 I_kwDOCGYnMM5DPcCD 404 Add example of `--convert` to the help for `sqlite-utils insert` simonw 9599 closed 0     2 2022-02-09T06:49:09Z 2022-02-09T06:56:35Z 2022-02-09T06:55:16Z OWNER  

https://sqlite-utils.datasette.io/en/3.23/cli-reference.html#insert would be more useful if it included an example of --convert in action.

I can maybe use an example from https://simonwillison.net/2022/Jan/11/sqlite-utils/

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/404/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1125081640 I_kwDOCGYnMM5DD2Io 401 Update SpatiaLite example in the documentation simonw 9599 closed 0     2 2022-02-06T02:02:07Z 2022-02-06T02:05:03Z 2022-02-06T02:03:24Z OWNER  

This one here: https://sqlite-utils.datasette.io/en/3.23/python-api.html#converting-column-values-using-sql-functions

It should take advantage of the new methods from: - #79

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/401/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
723460107 MDU6SXNzdWU3MjM0NjAxMDc= 187 Maybe: Utility method / CLI tool for initializing SpatiaLite simonw 9599 closed 0     2 2020-10-16T19:04:03Z 2022-02-05T00:04:26Z 2020-10-16T19:15:13Z OWNER  

I think this should initialize SpatiaLite against the current database if it has not been initialized already.

Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/187/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
557825032 MDU6SXNzdWU1NTc4MjUwMzI= 77 Ability to insert data that is transformed by a SQL function simonw 9599 closed 0     2 2020-01-30T23:45:55Z 2022-02-05T00:04:25Z 2020-01-31T00:24:32Z OWNER  

I want to be able to run the equivalent of this SQL insert: ```python

Convert to "Well Known Text" format

wkt = shape(geojson['geometry']).wkt

Insert and commit the record

conn.execute("INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))", ( "Wales", wkt )) conn.commit() ``` From the Datasette SpatiaLite docs: https://datasette.readthedocs.io/en/stable/spatialite.html

To do this, I need a way of telling sqlite-utils that a specific column should be wrapped in GeomFromText(?, 4326).

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/77/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1114638930 I_kwDOCGYnMM5CcApS 391 `sqlite-utils bulk` progress bar simonw 9599 closed 0     2 2022-01-26T05:14:49Z 2022-01-26T05:17:20Z 2022-01-26T05:16:51Z OWNER  

It can easily have a progress bar because it works by looping through an iterator: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/cli.py#L1014-L1018

Should also support the --silent option if I add this.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/391/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
471818939 MDU6SXNzdWU0NzE4MTg5Mzk= 48 Jupyter notebook demo of the library, launchable on Binder simonw 9599 closed 0     2 2019-07-23T17:05:05Z 2022-01-26T02:08:46Z 2022-01-26T02:08:39Z OWNER  
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/48/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1114544727 I_kwDOCGYnMM5CbppX 389 Plausible analytics for documentation simonw 9599 closed 0     2 2022-01-26T01:58:35Z 2022-01-26T02:07:41Z 2022-01-26T02:07:41Z OWNER  

```html

<script defer data-domain="sqlite-utils.datasette.io" src="https://plausible.io/js/plausible.js"></script>

```

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021785268

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/389/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1099584685 I_kwDOCGYnMM5BilSt 381 `sqlite-utils rows` options `--limit` and `--offset` simonw 9599 closed 0     2 2022-01-11T20:23:12Z 2022-01-11T23:33:37Z 2022-01-11T23:19:36Z OWNER  

Because I often want to use it just to preview a few rows from the database. Piping through | head -n 20 works for JSON and CSV (they stream) but not for --table.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/381/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097041471 PR_kwDOCGYnMM4wsVM6 367 Initial prototype of .analyze() methods simonw 9599 closed 0   3.21 7558727 2 2022-01-08T21:35:12Z 2022-01-10T19:31:08Z 2022-01-10T19:31:08Z OWNER simonw/sqlite-utils/pulls/367

Refs #366

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/367/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 1  
1066501534 I_kwDOCGYnMM4_kYWe 345 `table.strict` introspection boolean for identifying STRICT mode tables simonw 9599 closed 0     2 2021-11-29T21:05:10Z 2021-11-29T22:45:26Z 2021-11-29T22:44:36Z OWNER  

From the STRICT docs:

The SQLite parser accepts a comma-separated list of table options after the final close parenthesis in a CREATE TABLE statement. As of this writing (2021-08-23) only two options are recognized:

So I think I need to read the CREATE TABLE statement from the sqlite_master table, split on the last ), split those tokens on , and see if create is in there (case insensitive).

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982020757

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/345/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
991237645 MDExOlB1bGxSZXF1ZXN0NzI5NzMxNDQx 326 Test against 3.10-dev simonw 9599 closed 0     2 2021-09-08T15:01:15Z 2021-10-13T21:49:28Z 2021-10-13T21:49:28Z OWNER simonw/sqlite-utils/pulls/326

This tests against the latest 3.10 RC, https://www.python.org/downloads/release/python-3100rc2/

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/326/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
976405225 MDU6SXNzdWU5NzY0MDUyMjU= 320 sqlite-utils memory --analyze option simonw 9599 closed 0     2 2021-08-22T15:37:10Z 2021-08-22T15:46:56Z 2021-08-22T15:44:29Z OWNER  

To provide a way of running analyze-tables directly against JSON or CSV data.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/320/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
965210966 MDU6SXNzdWU5NjUyMTA5NjY= 314 Type signatures for `.create_table()` and `.create_table_sql()` and `.create()` and `Table.__init__` simonw 9599 closed 0     2 2021-08-10T18:03:59Z 2021-08-18T22:25:21Z 2021-08-18T22:25:21Z OWNER  

Adding type signatures to create_table() and .create_table_sql() is a bit too involved, I'll do that in a separate issue.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/312#issuecomment-896200682

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/314/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
974067156 MDU6SXNzdWU5NzQwNjcxNTY= 318 Research: handle gzipped CSV directly simonw 9599 open 0     2 2021-08-18T21:23:04Z 2021-08-18T21:25:30Z   OWNER  

Would it be worthwhile for the sqlite-utils command-line tool to grow features to efficiently directly interact with gzipped CSV data?

Maybe add --gz options to both insert and to the various commands that output query results.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/318/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
961008507 MDU6SXNzdWU5NjEwMDg1MDc= 308 Add an interactive tutorial as a Jupyter notebook simonw 9599 open 0     2 2021-08-04T20:34:22Z 2021-08-04T21:30:59Z   OWNER  

Can show people how to open this up in Binder.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/308/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
958516743 MDU6SXNzdWU5NTg1MTY3NDM= 306 Configure sphinx.ext.extlinks for issues simonw 9599 closed 0     2 2021-08-02T21:19:19Z 2021-08-02T21:39:34Z 2021-08-02T21:29:22Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/306/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
927766296 MDU6SXNzdWU5Mjc3NjYyOTY= 291 Adopt flake8 simonw 9599 closed 0     2 2021-06-23T01:19:37Z 2021-06-24T17:50:27Z 2021-06-24T17:50:27Z OWNER  
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/291/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
621286870 MDU6SXNzdWU2MjEyODY4NzA= 113 Syntactic sugar for ATTACH DATABASE simonw 9599 closed 0     2 2020-05-19T21:10:00Z 2021-02-19T05:09:12Z 2021-02-19T04:56:36Z OWNER  

https://www.sqlite.org/lang_attach.html

Maybe something like this: python db.attach("other_db", "other_db.db")

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/113/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
808028757 MDU6SXNzdWU4MDgwMjg3NTc= 231 limit=X, offset=Y parameters for more Python methods simonw 9599 closed 0     2 2021-02-14T19:31:23Z 2021-02-14T20:03:08Z 2021-02-14T20:03:08Z OWNER  

I'm going to add a offset= parameter to support this case. Thanks for the suggestion!

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/224#issuecomment-778828495

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/231/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
788527932 MDU6SXNzdWU3ODg1Mjc5MzI= 223 --delimiter option for CSV import simonw 9599 closed 0     2 2021-01-18T20:25:03Z 2021-02-06T01:39:47Z 2021-02-06T01:34:54Z OWNER  

https://bruxellesdata.opendatasoft.com/explore/dataset/dog-toilets/export/?location=12,50.85802,4.38054 says:

CSV uses semicolon (;) as a separator.

Would be useful to be able to do this:

sqlite-utils insert places.db places places.csv --delimiter ';'

--delimiter could imply --csv

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/223/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
738514367 MDU6SXNzdWU3Mzg1MTQzNjc= 202 sqlite-utils insert -f colname - for configuring full-text search simonw 9599 closed 0     2 2020-11-08T17:30:09Z 2021-01-03T05:00:36Z 2021-01-03T05:00:27Z OWNER  

A mechanism for specifying columns that should be configured for full-text search as part of the initial data import:

sqlite-utils insert mydb.db articles articles.csv --csv -f title -f body
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/202/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
777543336 MDU6SXNzdWU3Nzc1NDMzMzY= 217 Rename .escape() to .quote() simonw 9599 closed 0     2 2021-01-02T23:40:52Z 2021-01-03T04:27:38Z 2021-01-03T04:15:23Z OWNER  

.quote() is a better name because it reflects that the method adds quotes around the value.

This method has never been documented so I'm going to rename it without a major version bump.

sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/217/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
777529979 MDU6SXNzdWU3Nzc1Mjk5Nzk= 213 db.enable_counts() method simonw 9599 closed 0     2 2021-01-02T21:44:55Z 2021-01-02T22:04:02Z 2021-01-02T22:04:02Z OWNER  

Following #212 it would be useful if there was a utility method for enabling counts for ALL tables in a database:

python db.enable_counts()

Open question: should this setup triggers for virtual tables such as FTS tables? Could that break things?

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/213/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
706167456 MDU6SXNzdWU3MDYxNjc0NTY= 168 Automate (as much as possible) updates published to Homebrew simonw 9599 closed 0     2 2020-09-22T08:08:37Z 2020-11-09T07:43:30Z 2020-11-09T07:43:30Z OWNER  

I'd like to get new sqlite-utils (and Datasette) releases submitted to Homebrew as painlessly as possible.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/168/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
731740458 MDU6SXNzdWU3MzE3NDA0NTg= 191 Idea: @db.register_function(deterministic=True) simonw 9599 closed 0     2 2020-10-28T19:45:18Z 2020-10-28T21:31:06Z 2020-10-28T21:31:06Z OWNER  

Python 3.8 added a deterministic parameter to db.create_function(): https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function

sqlite-utils could expose this in the decorator, only actually applying it if the Python version supports it (using feature detection) - since nothing will break if it's not applied.

https://sqlite-utils.readthedocs.io/en/stable/python-api.html#registering-custom-sql-functions

```python db = Database(memory=True)

@db.register_function(deterministic=True) def reverse_string(s): return "".join(reversed(list(s))) ```

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/191/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
684118950 MDU6SXNzdWU2ODQxMTg5NTA= 138 extracts= doesn't configure foreign keys simonw 9599 closed 0     2 2020-08-23T05:21:15Z 2020-09-24T22:47:01Z 2020-09-24T22:46:52Z OWNER  

In using extracts= for shapefiles-to-sqlite in https://github.com/simonw/shapefile-to-sqlite/issues/9 I've run into a couple of pretty serious flaws:

  • The columns in the original table are still TEXT even when the foreign key they are supposed to reference is an INTEGER - which means Datasette foreign key features don't actually work
  • Those foreign key relationships aren't setup automatically - creating them is left as an exercise for the developer
 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/138/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
696045581 MDU6SXNzdWU2OTYwNDU1ODE= 155 rebuild-fts command and table.rebuild_fts() method simonw 9599 closed 0     2 2020-09-08T17:19:26Z 2020-09-24T20:35:46Z 2020-09-08T23:16:10Z OWNER  

https://sqlite.org/forum/forumpost/fa777fff86

Easiest thing would be to run a 'rebuild' to rebuild the FTS index from scratch based on the contents of the content table. i.e.

INSERT INTO licenses_fts(licenses_fts) VALUES('rebuild');

https://www.sqlite.org/fts5.html#the_rebuild_command

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/155/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
708293114 MDU6SXNzdWU3MDgyOTMxMTQ= 176 sqlite-utils transform column order option simonw 9599 closed 0     2 2020-09-24T16:01:21Z 2020-09-24T20:34:51Z 2020-09-24T16:11:59Z OWNER  

Split from #175

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/176/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
697179806 MDU6SXNzdWU2OTcxNzk4MDY= 157 sqlite-utils add-foreign-keys command simonw 9599 closed 0   2.19 5896742 2 2020-09-09T21:44:30Z 2020-09-24T20:34:50Z 2020-09-20T20:14:30Z OWNER  

Like add-foreign-key but can do multiple foreign keys at once. Inspired by https://github.com/simonw/calands-datasette/blob/99de39dd80a906f5c1f16724467b0cd55ba4ef36/build.sh which does this:

sqlite-utils add-foreign-key calands.db units_with_maps ACCESS_TYP sqlite-utils add-foreign-key calands.db units_with_maps AGNCY_NAME sqlite-utils add-foreign-key calands.db units_with_maps AGNCY_LEV sqlite-utils add-foreign-key calands.db units_with_maps AGNCY_TYP sqlite-utils add-foreign-key calands.db units_with_maps LAYER sqlite-utils add-foreign-key calands.db units_with_maps MNG_AGENCY sqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_LEV sqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_TYP sqlite-utils add-foreign-key calands.db units_with_maps COUNTY sqlite-utils add-foreign-key calands.db units_with_maps DES_TP

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/157/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
705995722 MDU6SXNzdWU3MDU5OTU3MjI= 162 A decorator for registering custom SQL functions simonw 9599 closed 0     2 2020-09-22T00:18:32Z 2020-09-22T00:40:44Z 2020-09-22T00:32:17Z OWNER  

Syntactic sugar for db.conn.create_function - it would work something like this:

```python db = sqlite_utils.Database("mydb.db")

@db.register_function def scramble(text): chars = list(text) random.shuffle(chars) return "".join(chars) The decorator would inspect the function to find its name and arity (number of arguments). Having run the above you could then do:python db.execute("select scramble('hello')").fetchall() ```

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/162/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
695441530 MDU6SXNzdWU2OTU0NDE1MzA= 154 OperationalError: cannot change into wal mode from within a transaction simonw 9599 open 0     2 2020-09-07T23:42:44Z 2020-09-07T23:47:10Z   OWNER  

I'm getting this error when running:

sqlite-utils enable-wal beta.db

OperationalError: cannot change into wal mode from within a transaction

I'm worried that maybe that's because of this new code from #152:

https://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/sqlite_utils/db.py#L128-L129

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/154/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
695376054 MDU6SXNzdWU2OTUzNzYwNTQ= 152 Turn on recursive_triggers by default simonw 9599 closed 0     2 2020-09-07T20:26:36Z 2020-09-07T21:17:48Z 2020-09-07T20:45:14Z OWNER  

https://www.sqlite.org/pragma.html#pragma_recursive_triggers says:

Prior to SQLite version 3.6.18 (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite.

So I think the fix for the complex issue in #149 is to turn on recursive_triggers globally by default for sqlite-utils.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/152/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
688395275 MDU6SXNzdWU2ODgzOTUyNzU= 144 Run some tests against numpy simonw 9599 closed 0     2 2020-08-28T22:53:00Z 2020-08-28T22:57:05Z 2020-08-28T22:57:04Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/144/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
668308777 MDU6SXNzdWU2NjgzMDg3Nzc= 129 "insert-files --sqlar" for creating SQLite archives simonw 9599 closed 0     2 2020-07-30T02:28:29Z 2020-07-30T22:41:01Z 2020-07-30T22:40:55Z OWNER  

A --sqlar option could cause insert-files to behave in the same way as SQLite's own sqlar mechanism.

https://www.sqlite.org/sqlar.html and https://sqlite.org/sqlar/doc/trunk/README.md

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/129/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
665819048 MDU6SXNzdWU2NjU4MTkwNDg= 126 Ability to insert binary data on the CLI using JSON simonw 9599 closed 0     2 2020-07-26T16:54:14Z 2020-07-27T04:00:33Z 2020-07-27T03:59:45Z OWNER  

I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says "these columns are base64 encoded, store the decoded data in a BLOB".

That would solve inserting binary data using JSON too. Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/126/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
644122661 MDU6SXNzdWU2NDQxMjI2NjE= 116 Documentation for table.pks introspection property simonw 9599 closed 0     2 2020-06-23T20:27:24Z 2020-06-23T21:21:33Z 2020-06-23T21:03:14Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/116/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
615477131 MDU6SXNzdWU2MTU0NzcxMzE= 111 sqlite-utils drop-table and drop-view commands simonw 9599 closed 0     2 2020-05-10T21:10:42Z 2020-05-11T01:58:36Z 2020-05-11T00:44:26Z OWNER  

Would be useful to be able to drop views and tables from the CLI.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/111/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
611222968 MDU6SXNzdWU2MTEyMjI5Njg= 107 sqlite-utils create-view CLI command simonw 9599 closed 0     2 2020-05-02T16:15:13Z 2020-05-03T15:36:58Z 2020-05-03T15:36:37Z OWNER  

Can go with #27 - sqlite-utils create-table.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/107/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
611326701 MDU6SXNzdWU2MTEzMjY3MDE= 108 Documentation unit tests for CLI commands simonw 9599 closed 0     2 2020-05-03T03:58:42Z 2020-05-03T04:13:57Z 2020-05-03T04:13:57Z OWNER  

Have a test that ensures all CLI commands are documented.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/108/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
598640234 MDU6SXNzdWU1OTg2NDAyMzQ= 99 .upsert_all() should maybe error if dictionaries passed to it do not have the same keys simonw 9599 closed 0     2 2020-04-13T03:02:25Z 2020-04-13T03:05:20Z 2020-04-13T03:05:04Z OWNER  

While investigating #98 I stumbled across this: ``` def test_upsert_compound_primary_key(fresh_db): table = fresh_db["table"] table.upsert_all( [ {"species": "dog", "id": 1, "name": "Cleo", "age": 4}, {"species": "cat", "id": 1, "name": "Catbag"}, ], pk=("species", "id"), ) table.upsert_all( [ {"species": "dog", "id": 1, "age": 5}, {"species": "dog", "id": 2, "name": "New Dog", "age": 1}, ], pk=("species", "id"), )

  assert [
        {"species": "dog", "id": 1, "name": "Cleo", "age": 5},
        {"species": "cat", "id": 1, "name": "Catbag", "age": None},
        {"species": "dog", "id": 2, "name": "New Dog", "age": 1},
    ] == list(table.rows)

E AssertionError: assert [{'age': 5, '...cies': 'dog'}] == [{'age': 5, '...cies': 'dog'}] E At index 0 diff: {'species': 'dog', 'id': 1, 'name': 'Cleo', 'age': 5} != {'species': 'dog', 'id': 1, 'name': None, 'age': 5} E Full diff: E - [{'age': 5, 'id': 1, 'name': 'Cleo', 'species': 'dog'}, E ? ^^^ -- E + [{'age': 5, 'id': 1, 'name': None, 'species': 'dog'}, E ? ^^^ E {'age': None, 'id': 1, 'name': 'Catbag', 'species': 'cat'}, E {'age': 1, 'id': 2, 'name': 'New Dog', 'species': 'dog'}] `` If you run.upsert_all()` with multiple dictionaries it doesn't quite have the effect you might expect.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/99/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
565837965 MDU6SXNzdWU1NjU4Mzc5NjU= 87 Should detect collections.OrderedDict as a regular dictionary simonw 9599 closed 0     2 2020-02-16T02:06:34Z 2020-02-16T02:20:59Z 2020-02-16T02:20:59Z OWNER  

File "...python3.7/site-packages/sqlite_utils/db.py", line 292, in create_table column_type=COLUMN_TYPE_MAPPING[column_type], KeyError: <class 'collections.OrderedDict'>

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/87/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
529376481 MDExOlB1bGxSZXF1ZXN0MzQ2MjY0OTI2 67 Run tests against 3.5 too simonw 9599 closed 0     2 2019-11-27T14:20:35Z 2019-12-31T01:29:44Z 2019-12-31T01:29:43Z OWNER simonw/sqlite-utils/pulls/67
sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/67/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
462430920 MDU6SXNzdWU0NjI0MzA5MjA= 35 table.update(...) method simonw 9599 closed 0     2 2019-06-30T18:06:15Z 2019-07-28T15:43:52Z 2019-07-28T15:43:52Z OWNER  

Spun off from #23 - this method will allow a user to update a specific row.

Currently the only way to do that it is to call .upsert({full record}) with the primary key field matching an existing record - but this does not support partial updates. python db["events"].update(3, {"name": "Renamed"}) This method only works on an existing table, so there's no need for a pk="id" specifier - it can detect the primary key by looking at the table.

If the primary key is compound the first argument can be a tuple: python db["events_venues"].update((3, 2), {"custom_label": "Label"}) The method can be called without the second dictionary argument. Doing this selects the row specified by the primary key (throwing an error if it does not exist) and remembers it so that chained operations can be carried out - see proposal in https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507055345

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/35/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
467862459 MDExOlB1bGxSZXF1ZXN0Mjk3NDEyNDY0 38 table.update() method simonw 9599 closed 0     2 2019-07-14T17:03:49Z 2019-07-28T15:43:51Z 2019-07-28T15:43:51Z OWNER simonw/sqlite-utils/pulls/38

Refs #35

Still to do:

  • [x] Unit tests
  • [x] Switch to using .get()
  • [x] Better exceptions, plus unit tests for what happens if pk does not exist
  • [x] Documentation
  • [x] Ensure compound primary keys work properly
  • [x] alter=True support
 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/38/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
471628483 MDU6SXNzdWU0NzE2Mjg0ODM= 44 Utilities for building lookup tables simonw 9599 closed 0     2 2019-07-23T10:59:58Z 2019-07-23T13:07:01Z 2019-07-23T13:07:01Z OWNER  

While building https://github.com/dogsheep/healthkit-to-sqlite I found a need for a neat mechanism for easily building lookup tables - tables where each unique value in a column is replaced by a foreign key to a separate table.

csvs-to-sqlite currently creates those with its "extract" mechanism - but that's written as custom code against Pandas. I'd like to eventually replace Pandas with sqlite-utils there.

See also #42

sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/44/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
462423839 MDU6SXNzdWU0NjI0MjM4Mzk= 33 index_foreign_keys / index-foreign-keys utilities simonw 9599 closed 0     2 2019-06-30T16:42:03Z 2019-06-30T23:54:11Z 2019-06-30T23:50:55Z OWNER  

Sometimes it's good to have indices on all columns that are foreign keys, to allow for efficient reverse lookups.

This would be a useful utility:

$ sqlite-utils index-foreign-keys database.db
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/33/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
448395665 MDU6SXNzdWU0NDgzOTU2NjU= 22 Release notes for 1.0 simonw 9599 closed 0   1.0 4348046 2 2019-05-25T00:58:03Z 2019-05-25T01:18:27Z 2019-05-25T01:06:52Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/22/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
413871266 MDU6SXNzdWU0MTM4NzEyNjY= 18 .insert/.upsert/.insert_all/.upsert_all should add missing columns simonw 9599 closed 0   1.0 4348046 2 2019-02-24T21:36:11Z 2019-05-25T00:42:11Z 2019-05-25T00:42:11Z OWNER  

This is a larger change, but it would be incredibly useful: if you attempt to insert or update a document with a field that does not currently exist in the underlying table, sqlite-utils should add the appropriate column for you.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/18/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
413740684 MDU6SXNzdWU0MTM3NDA2ODQ= 11 Detect numpy types when creating tables simonw 9599 closed 0     2 2019-02-23T21:09:35Z 2019-02-24T04:02:20Z 2019-02-24T04:02:20Z OWNER  

Inspired by #8

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/11/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);