html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553528386,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553528386,MDEyOklzc3VlQ29tbWVudDU1MzUyODM4Ng==,9599,2019-11-13T18:03:10Z,2019-11-13T18:03:54Z,OWNER,"Maybe `inplace()` (combining ""insert"" and ""replace"")?

It could be an alias for `.insert(..., replace=True)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,
https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553528850,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553528850,MDEyOklzc3VlQ29tbWVudDU1MzUyODg1MA==,9599,2019-11-13T18:04:20Z,2019-11-13T18:04:20Z,OWNER,This is going to affect the design of the CLI subcommands as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,
https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553540146,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553540146,MDEyOklzc3VlQ29tbWVudDU1MzU0MDE0Ng==,9599,2019-11-13T18:33:30Z,2019-11-13T18:33:30Z,OWNER,"Maybe instead of inventing a new term I should tell people to use `.insert(..., replace=True)` directly. That matches `ignore=True`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,
https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553574011,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553574011,MDEyOklzc3VlQ29tbWVudDU1MzU3NDAxMQ==,9599,2019-11-13T19:53:45Z,2019-11-13T19:53:45Z,OWNER,"First step: add a `replace=True` argument to `insert()` and `insert_all()` that does the same thing as the current `upsert=True`

https://github.com/simonw/sqlite-utils/blob/8dab9fd1ccf571e188eec9ccf606a0c50fccf200/sqlite_utils/db.py#L938-L946","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,
https://github.com/simonw/sqlite-utils/issues/66#issuecomment-554565198,https://api.github.com/repos/simonw/sqlite-utils/issues/66,554565198,MDEyOklzc3VlQ29tbWVudDU1NDU2NTE5OA==,9599,2019-11-15T23:12:28Z,2019-11-15T23:12:28Z,OWNER,"Urgh this is going to be quite a bit of work, especially in the CLI module which shares an implementation for `upsert` and `insert` in a way that looks like it will have to be unwrapped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,
https://github.com/simonw/sqlite-utils/issues/66#issuecomment-555690319,https://api.github.com/repos/simonw/sqlite-utils/issues/66,555690319,MDEyOklzc3VlQ29tbWVudDU1NTY5MDMxOQ==,9599,2019-11-19T20:10:17Z,2019-11-19T20:10:17Z,OWNER,"Thinking about this further: I believe every time I've personally used `upsert` in the past (either with the Python library or the CLI tool) I've actually wanted the new behaviour, where ""upsert"" means ""update existing record with these changes, or insert a new record if one does not exist"".

So I'm happy with `upsert` doing that, and `insert --replace` being added as an option that does what `upsert` does ta the moment.

I'll still ship it as version 2.0 since it's technically a breaking change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,
https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569131397,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569131397,MDEyOklzc3VlQ29tbWVudDU2OTEzMTM5Nw==,9599,2019-12-26T20:49:11Z,2019-12-26T20:49:11Z,OWNER,Don't forget to update the documentation. This will be quite an involved task.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,
https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569226620,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569226620,MDEyOklzc3VlQ29tbWVudDU2OTIyNjYyMA==,9599,2019-12-27T09:05:29Z,2019-12-27T09:05:36Z,OWNER,"I'm going to start by ignoring the existing `upsert` entirely and implementing `.insert(..., replace=True)` and `$ sqlite-utils insert --replace`. Including updating the tests.

Then I'll figure out how to implement the new `.upsert()` / `$ sqlite-utils upsert`.

Then I'll update the documentation, and ship `sqlite-utils` 2.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,
https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569588216,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569588216,MDEyOklzc3VlQ29tbWVudDU2OTU4ODIxNg==,9599,2019-12-30T05:31:45Z,2019-12-30T05:31:45Z,OWNER,Last step: update changelog and ship 2.0. Then I can close this issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,
https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569844426,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569844426,MDEyOklzc3VlQ29tbWVudDU2OTg0NDQyNg==,9599,2019-12-31T01:30:20Z,2019-12-31T01:30:20Z,OWNER,"I shipped 2.0 - release notes here: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v2

I also wrote about it on my blog: https://simonwillison.net/2019/Dec/30/sqlite-utils-2/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,
https://github.com/simonw/sqlite-utils/issues/63#issuecomment-549430429,https://api.github.com/repos/simonw/sqlite-utils/issues/63,549430429,MDEyOklzc3VlQ29tbWVudDU0OTQzMDQyOQ==,9599,2019-11-04T16:20:35Z,2019-11-04T16:20:35Z,OWNER,"I don't think we need this. We already have a `dogs.create_index([""name""], if_not_exists=True)` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517241040,
https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549418134,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549418134,MDEyOklzc3VlQ29tbWVudDU0OTQxODEzNA==,9599,2019-11-04T15:54:18Z,2019-11-04T15:54:18Z,OWNER,"This is a good idea. The individual row version of it can work like the `.update(pk, ...)` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,
https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549425012,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549425012,MDEyOklzc3VlQ29tbWVudDU0OTQyNTAxMg==,9599,2019-11-04T16:09:08Z,2019-11-04T16:09:08Z,OWNER,Documentation for `table.delete()`: https://github.com/simonw/sqlite-utils/blob/19073d6d972fad9d68dd74c28544cd29083f1c12/docs/python-api.rst#deleting-a-specific-record,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,
https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549425364,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549425364,MDEyOklzc3VlQ29tbWVudDU0OTQyNTM2NA==,9599,2019-11-04T16:09:56Z,2019-11-04T16:09:56Z,OWNER,"Since we have `table.rows_where(where, where_args)` it makes sense to me to also support `table.delete_where(where, where_args)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,
https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549429512,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549429512,MDEyOklzc3VlQ29tbWVudDU0OTQyOTUxMg==,9599,2019-11-04T16:18:48Z,2019-11-04T16:18:48Z,OWNER,Documentation for `.delete_where()`: https://github.com/simonw/sqlite-utils/blob/169ea455fc1f1d5e5b6e44cb339ba7ffa9d49c31/docs/python-api.rst#deleting-multiple-records,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,
https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549435364,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549435364,MDEyOklzc3VlQ29tbWVudDU0OTQzNTM2NA==,9599,2019-11-04T16:30:34Z,2019-11-04T16:30:34Z,OWNER,Released as 1.12.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,
https://github.com/simonw/sqlite-utils/issues/61#issuecomment-533818697,https://api.github.com/repos/simonw/sqlite-utils/issues/61,533818697,MDEyOklzc3VlQ29tbWVudDUzMzgxODY5Nw==,49260,2019-09-21T18:09:01Z,2019-09-21T18:09:28Z,CONTRIBUTOR,"@witeshadow The library version doesn't have helpers around CSV (at least not from what I can see in the code). 

But here's a snippet that makes it easy to insert from CSV using the library. 

```
import csv
from sqlite_utils import Database

# CSV Reader

csv_file = open(""filename.csv"")   # open the csv file.
reader = csv.reader(csv_file)  # Create a CSV reader
headers = next(reader)   # First line is the header
docs = (dict(zip(headers, row)) for row in reader)

# Now you can use the `sqlite_utils` library. 

db = Database(""my_database.db"")
db[""table_name""].insert_all(docs)
```

This snippet is adapted from reading the CLI source code on how it implements the csv option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",491219910,
https://github.com/simonw/sqlite-utils/issues/61#issuecomment-549432592,https://api.github.com/repos/simonw/sqlite-utils/issues/61,549432592,MDEyOklzc3VlQ29tbWVudDU0OTQzMjU5Mg==,9599,2019-11-04T16:25:01Z,2019-11-04T16:25:01Z,OWNER,"Yeah I've thought about this a bit and I'm OK leaving it out.

The core idea of `sqlite-utils` is that if you can create a list, iterator or generator of Python dictionaries you can efficiently insert those into a SQLite table.

The `--csv` function is actually implemented as just a few lines of code which turn that incoming CSV into a generator of dictionaries:

https://github.com/simonw/sqlite-utils/blob/169ea455fc1f1d5e5b6e44cb339ba7ffa9d49c31/sqlite_utils/cli.py#L364-L368

I could turn this into a reusable helper function but since it wouldn't have anything to do with database inserts (it would just be a helper that turns a CSV into a generator of dictionaries) it doesn't feel like it fits well in the Python library section of this package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",491219910,
https://github.com/simonw/sqlite-utils/issues/606#issuecomment-1843465748,https://api.github.com/repos/simonw/sqlite-utils/issues/606,1843465748,IC_kwDOCGYnMM5t4Q4U,9599,2023-12-06T18:36:51Z,2023-12-06T18:36:51Z,OWNER,I'll add `bytes` too - `float` already works. This makes sense because when you are working with the Python API you use `str` and `float` and `bytes` and `int` to specify column types.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2029161033,
https://github.com/simonw/sqlite-utils/issues/606#issuecomment-1843579184,https://api.github.com/repos/simonw/sqlite-utils/issues/606,1843579184,IC_kwDOCGYnMM5t4skw,9599,2023-12-06T19:43:55Z,2023-12-06T19:43:55Z,OWNER,"Updated documentation:
- https://sqlite-utils.datasette.io/en/latest/cli.html#cli-add-column
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2029161033,
https://github.com/simonw/sqlite-utils/issues/605#issuecomment-1846554637,https://api.github.com/repos/simonw/sqlite-utils/issues/605,1846554637,IC_kwDOCGYnMM5uEDAN,9599,2023-12-08T05:07:54Z,2023-12-08T05:07:54Z,OWNER,"Thanks for opening an issue - this should help future Google searchers figure out what's going on here.

Another approach here could be to store large integers as `TEXT` in SQLite (or even as `BLOB`).

Both storing as `REAL` and storing as `TEXT/BLOB` feel nasty to me, but it looks like SQLite has a hard upper limit of 9223372036854775807 for integers.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2007893839,
https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843585454,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843585454,IC_kwDOCGYnMM5t4uGu,22429695,2023-12-06T19:48:26Z,2023-12-08T05:05:03Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`9286c1b`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/9286c1ba432e890b1bb4b2a1f847b15364c1fa18?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`1698a9d`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.72%.
> Report is 1 commits behind head on main.

> :exclamation: Current head 1698a9d differs from pull request most recent head 61c6e26. Consider uploading reports for the commit 61c6e26 to get more accurate results


<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #604      +/-   ##
==========================================
- Coverage   95.77%   95.72%   -0.06%     
==========================================
  Files           8        8              
  Lines        2842     2852      +10     
==========================================
+ Hits         2722     2730       +8     
- Misses        120      122       +2     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,"{""id"": 254, ""slug"": ""codecov"", ""node_id"": ""MDM6QXBwMjU0"", ""owner"": {""login"": ""codecov"", ""id"": 8226205, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/8226205?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/codecov"", ""html_url"": ""https://github.com/codecov"", ""followers_url"": ""https://api.github.com/users/codecov/followers"", ""following_url"": ""https://api.github.com/users/codecov/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/codecov/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/codecov/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/codecov/subscriptions"", ""organizations_url"": ""https://api.github.com/users/codecov/orgs"", ""repos_url"": ""https://api.github.com/users/codecov/repos"", ""events_url"": ""https://api.github.com/users/codecov/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/codecov/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Codecov"", ""description"": ""Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. "", ""external_url"": ""https://codecov.io"", ""html_url"": ""https://github.com/apps/codecov"", ""created_at"": ""2016-09-25T14:18:27Z"", ""updated_at"": ""2023-09-08T15:29:16Z"", ""permissions"": {""administration"": ""read"", ""checks"": ""write"", ""contents"": ""read"", ""emails"": ""read"", ""issues"": ""read"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""write""}, ""events"": [""check_run"", ""check_suite"", ""create"", ""delete"", ""fork"", ""member"", ""membership"", ""organization"", ""public"", ""pull_request"", ""push"", ""release"", ""repository"", ""status"", ""team_add""]}"
https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843586503,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843586503,IC_kwDOCGYnMM5t4uXH,9599,2023-12-06T19:49:10Z,2023-12-06T19:49:29Z,OWNER,"This looks really great on first glance - design is good, implementation is solid, tests and documentation look great.

Looks like a couple of `mypy` failures in the tests at the moment:

```
  mypy sqlite_utils tests

sqlite_utils/db.py:543: error: Incompatible types in assignment (expression has type ""type[Table]"", variable has type ""type[View]"")  [assignment]
tests/test_lookup.py:156: error: Name ""test_lookup_new_table"" already defined on line 5  [no-redef]
Found 2 errors in 2 files (checked 54 source files)
Error: Process completed with exit code 1.
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,
https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843975536,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843975536,IC_kwDOCGYnMM5t6NVw,16437338,2023-12-07T01:17:05Z,2023-12-07T01:17:05Z,CONTRIBUTOR,Apologies - I pushed a fix that addresses the mypy failures.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,
https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1846560096,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1846560096,IC_kwDOCGYnMM5uEEVg,9599,2023-12-08T05:16:44Z,2023-12-08T05:17:20Z,OWNER,"Also tested this manually like so:

```bash
sqlite-utils create-table strict.db strictint id integer size integer --strict
sqlite-utils create-table strict.db notstrictint id integer size integer         
sqlite-utils install sqlite-utils-shell
sqlite-utils shell strict.db                                             
```
```
Attached to strict.db
Type 'exit' to exit.
sqlite-utils> insert into strictint (size) values (4);
1 row affected
sqlite-utils> insert into strictint (size) values ('four');
An error occurred: cannot store TEXT value in INTEGER column strictint.size
sqlite-utils> insert into notstrictint (size) values ('four');
1 row affected
sqlite-utils> commit;
Done
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,
https://github.com/simonw/sqlite-utils/issues/603#issuecomment-1846555822,https://api.github.com/repos/simonw/sqlite-utils/issues/603,1846555822,IC_kwDOCGYnMM5uEDSu,9599,2023-12-08T05:09:55Z,2023-12-08T05:10:31Z,OWNER,"I'm unable to replicate this issue. This is with a fresh install of `sqlite-utils==3.35.2`:
```
(base) ~ python3.12
Python 3.12.0 (v3.12.0:0fb18b02c8, Oct  2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type ""help"", ""copyright"", ""credits"" or ""license"" for more information.
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> db[""foo""].insert({""bar"": 1})
<Table foo (bar)>
>>> import sys
>>> sys.version
'3.12.0 (v3.12.0:0fb18b02c8, Oct  2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)]'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1988525411,
https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793263638,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793263638,IC_kwDOCGYnMM5q4wgW,9599,2023-11-04T00:19:58Z,2023-11-04T00:19:58Z,OWNER,Thanks for this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,
https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793264654,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793264654,IC_kwDOCGYnMM5q4wwO,22429695,2023-11-04T00:22:07Z,2023-11-04T00:27:29Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`622c3a5`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/622c3a5a7dd53a09c029e2af40c2643fe7579340?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b1a6076`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #600   +/-   ##
=======================================
  Coverage   95.77%   95.77%           
=======================================
  Files           8        8           
  Lines        2840     2840           
=======================================
  Hits         2720     2720           
  Misses        120      120           
```


| [Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <ø> (ø)` | |
| [sqlite\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <ø> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,"{""id"": 254, ""slug"": ""codecov"", ""node_id"": ""MDM6QXBwMjU0"", ""owner"": {""login"": ""codecov"", ""id"": 8226205, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/8226205?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/codecov"", ""html_url"": ""https://github.com/codecov"", ""followers_url"": ""https://api.github.com/users/codecov/followers"", ""following_url"": ""https://api.github.com/users/codecov/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/codecov/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/codecov/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/codecov/subscriptions"", ""organizations_url"": ""https://api.github.com/users/codecov/orgs"", ""repos_url"": ""https://api.github.com/users/codecov/repos"", ""events_url"": ""https://api.github.com/users/codecov/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/codecov/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Codecov"", ""description"": ""Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. "", ""external_url"": ""https://codecov.io"", ""html_url"": ""https://github.com/apps/codecov"", ""created_at"": ""2016-09-25T14:18:27Z"", ""updated_at"": ""2023-09-08T15:29:16Z"", ""permissions"": {""administration"": ""read"", ""checks"": ""write"", ""contents"": ""read"", ""emails"": ""read"", ""issues"": ""read"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""write""}, ""events"": [""check_run"", ""check_suite"", ""create"", ""delete"", ""fork"", ""member"", ""membership"", ""organization"", ""public"", ""pull_request"", ""push"", ""release"", ""repository"", ""status"", ""team_add""]}"
https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793265952,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793265952,IC_kwDOCGYnMM5q4xEg,9599,2023-11-04T00:25:34Z,2023-11-04T00:25:34Z,OWNER,The tests failed because they found a spelling mistake in a completely unrelated area of the code - not sure why that had not been caught before.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,
https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793268126,IC_kwDOCGYnMM5q4xme,9599,2023-11-04T00:31:34Z,2023-11-04T00:31:34Z,OWNER,"Testing this manually on macOS using Docker Desk top like this:
```bash
docker run -it --rm arm64v8/ubuntu /bin/bash
```
Then inside the container:
```bash
uname -m
```
Outputs: `aarch64`

Then:
```bash
apt install spatialite-bin libsqlite3-mod-spatialite git python3 python3-venv -y
cd /tmp
git clone https://github.com/simonw/sqlite-utils
cd sqlite-utils
python3 -m venv venv
source venv/bin/activate
pip install -e '.[test]'
sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite
```
Which output:
```
Traceback (most recent call last):
  File ""/tmp/sqlite-utils/venv/bin/sqlite-utils"", line 33, in <module>
    sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1157, in __call__
    return self.main(*args, **kwargs)
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1078, in main
    rv = self.invoke(ctx)
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 783, in invoke
    return __callback(*args, **kwargs)
  File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 1959, in memory
    _load_extensions(db, load_extension)
  File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 3232, in _load_extensions
    if "":"" in ext:
TypeError: argument of type 'NoneType' is not iterable
```
Then I ran this:
```bash
git checkout -b MikeCoats-spatialite-paths-linux-arm main
git pull https://github.com/MikeCoats/sqlite-utils.git spatialite-paths-linux-arm
```
And now:
```bash
sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite
```
Outputs:
```json
[{""spatialite_version()"": ""5.0.1""}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,
https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793269219,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793269219,IC_kwDOCGYnMM5q4x3j,9599,2023-11-04T00:34:33Z,2023-11-04T00:34:33Z,OWNER,"The GIS tests now pass in that container too:
```bash
pytest tests/test_gis.py
```
```
======================== test session starts =========================
platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/sqlite-utils
plugins: hypothesis-6.88.1
collected 12 items                                                   

tests/test_gis.py ............                                 [100%]

========================= 12 passed in 0.48s =========================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,
https://github.com/simonw/sqlite-utils/issues/6#issuecomment-457978729,https://api.github.com/repos/simonw/sqlite-utils/issues/6,457978729,MDEyOklzc3VlQ29tbWVudDQ1Nzk3ODcyOQ==,9599,2019-01-28T02:12:19Z,2019-01-28T02:12:19Z,OWNER,Will need to solve #7 for this to become truly efficient.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403624090,
https://github.com/simonw/sqlite-utils/issues/599#issuecomment-1793268750,https://api.github.com/repos/simonw/sqlite-utils/issues/599,1793268750,IC_kwDOCGYnMM5q4xwO,9599,2023-11-04T00:33:25Z,2023-11-04T00:33:25Z,OWNER,"See details of how I tested this here:
- https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126

Short version: having applied this fix, the following command (on simulated `aarch64`):
```bash
sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite
```
Outputs:
```json
[{""spatialite_version()"": ""5.0.1""}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1976986318,
https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793272429,https://api.github.com/repos/simonw/sqlite-utils/issues/598,1793272429,IC_kwDOCGYnMM5q4ypt,9599,2023-11-04T00:40:34Z,2023-11-04T00:40:34Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1926729132,
https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793274485,https://api.github.com/repos/simonw/sqlite-utils/issues/598,1793274485,IC_kwDOCGYnMM5q4zJ1,9599,2023-11-04T00:46:55Z,2023-11-04T00:46:55Z,OWNER,"Manually tested. Before:

![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/3bdd30ea-1a54-4fec-b42d-793130a17bc1)

After:

![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/015d4e4e-b40c-4a93-81f5-1a4adef69b11)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1926729132,
https://github.com/simonw/sqlite-utils/pull/596#issuecomment-1793274869,https://api.github.com/repos/simonw/sqlite-utils/issues/596,1793274869,IC_kwDOCGYnMM5q4zP1,9599,2023-11-04T00:47:55Z,2023-11-04T00:47:55Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1919296686,
https://github.com/simonw/sqlite-utils/issues/595#issuecomment-1733312349,https://api.github.com/repos/simonw/sqlite-utils/issues/595,1733312349,IC_kwDOCGYnMM5nUD9d,123451970,2023-09-25T09:38:13Z,2023-09-25T09:38:57Z,NONE,"Never mind

When I created the connection using 
`sqlite_utils.Database(path)`

I just needed to add the following statement right after and it did the trick

`self.db.conn.execute(""PRAGMA foreign_keys = ON"")`

Hope this helps people in the future 👍 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907281675,
https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714919806,https://api.github.com/repos/simonw/sqlite-utils/issues/594,1714919806,IC_kwDOCGYnMM5mN5l-,9599,2023-09-12T03:49:41Z,2023-09-12T03:49:41Z,OWNER,"Digging in a bit more:
```pycon
>>> pprint(list(db.query('PRAGMA foreign_key_list(courses)')))
[{'from': 'campus_name',
  'id': 0,
  'match': 'NONE',
  'on_delete': 'NO ACTION',
  'on_update': 'NO ACTION',
  'seq': 0,
  'table': 'departments',
  'to': 'campus_name'},
 {'from': 'dept_code',
  'id': 0,
  'match': 'NONE',
  'on_delete': 'NO ACTION',
  'on_update': 'NO ACTION',
  'seq': 1,
  'table': 'departments',
  'to': 'dept_code'}]
```
I think the way you tell it's a compound foreign key is that both of those have the same `id` value - of `0` - but they then have two different `seq` values of `0` and `1`.

Right now I ignore those columns entirely: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L1523-L1540","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891614971,
https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714920708,https://api.github.com/repos/simonw/sqlite-utils/issues/594,1714920708,IC_kwDOCGYnMM5mN50E,9599,2023-09-12T03:51:13Z,2023-09-12T03:51:13Z,OWNER,"Changing this without breaking backwards compatibility (and forcing a 4.0 release) will be tricky, because `ForeignKey()` is a `namedtuple`:

https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L148-L150

I could swap it out for a `dataclass` and add those extra columns, but I need to make sure that code like this still works:
```python
for table, column, other_table, other_column in table.foreign_keys:
   # ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891614971,
https://github.com/simonw/sqlite-utils/pull/593#issuecomment-1710939868,https://api.github.com/repos/simonw/sqlite-utils/issues/593,1710939868,IC_kwDOCGYnMM5l-t7c,22429695,2023-09-08T01:03:40Z,2023-09-09T00:44:52Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and no project coverage change.
> Comparison is base [(`5d123f0`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/5d123f031fc4fadc98f508e0ef6b7b6671e86155?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b86374f`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #593   +/-   ##
=======================================
  Coverage   95.77%   95.77%           
=======================================
  Files           8        8           
  Lines        2837     2840    +3     
=======================================
+ Hits         2717     2720    +3     
  Misses        120      120           
```


| [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <100.00%> (+<0.01%)` | :arrow_up: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886783150,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710930934,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710930934,IC_kwDOCGYnMM5l-rv2,9599,2023-09-08T00:47:57Z,2023-09-08T00:47:57Z,OWNER,"That's odd, I wrote a test for this just now and it passes already:
```python
def test_transform_preserves_rowids(fresh_db):
    # Create a rowid table
    fresh_db[""places""].insert_all(
        (
            {""name"": ""Paris"", ""country"": ""France""},
            {""name"": ""London"", ""country"": ""UK""},
            {""name"": ""New York"", ""country"": ""USA""},
        ),
    )
    assert fresh_db[""places""].use_rowid
    previous_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, name from places"")
    )
    # Transform it
    fresh_db[""places""].transform(column_order=(""country"", ""name""))
    # Should be the same
    next_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, name from places"")
    )
    assert previous_rows == next_rows
```
So maybe I'm wrong about the cause of that bug?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710931605,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710931605,IC_kwDOCGYnMM5l-r6V,9599,2023-09-08T00:49:02Z,2023-09-08T00:49:02Z,OWNER,"I tried bumping that up to 10,000 rows instead of just 3 but the test still passed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710933716,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710933716,IC_kwDOCGYnMM5l-sbU,9599,2023-09-08T00:52:42Z,2023-09-08T00:52:42Z,OWNER,"I just noticed that the table where I encountered this bug wasn't actually a `rowid` table after all - it had an `id` column that was a text primary key.

The reason the `rowid` was important is that's how the FTS mechanism in Datasette relates FTS entries to their rows.

But I tried this test and it passed, too:
```python
def test_transform_preserves_rowids(fresh_db):
    fresh_db[""places""].insert_all(
        [
            {""id"": ""1"", ""name"": ""Paris"", ""country"": ""France""},
            {""id"": ""2"", ""name"": ""London"", ""country"": ""UK""},
            {""id"": ""3"", ""name"": ""New York"", ""country"": ""USA""},
        ],
        pk=""id"",
    )
    previous_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"")
    )
    # Transform it
    fresh_db[""places""].transform(column_order=(""country"", ""name""))
    # Should be the same
    next_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"")
    )
    assert previous_rows == next_rows
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710934448,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710934448,IC_kwDOCGYnMM5l-smw,9599,2023-09-08T00:54:03Z,2023-09-08T00:54:03Z,OWNER,"Oh! Maybe the row ID preservation here is a coincidence because the tables are created from scratch and count 1, 2, 3.

If I delete a row from the table and then insert some more - breaking the `rowid` sequence - it might show the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710935270,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710935270,IC_kwDOCGYnMM5l-szm,9599,2023-09-08T00:55:30Z,2023-09-08T00:55:30Z,OWNER,"Yes! That recreated the bug:
```
>       assert previous_rows == next_rows
E       AssertionError: assert equals failed
E         [                                                                [                                                               
E           (1, '1', 'Paris'),                                               (1, '1', 'Paris'),                                            
E           (3, '3', 'New York'),                                            (2, '3', 'New York'),                                         
E           (4, '4', 'London'),                                              (3, '4', 'London'),                                           
E         ]                                                       ...
E         
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1712895580,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1712895580,IC_kwDOCGYnMM5mGLZc,9599,2023-09-10T17:46:41Z,2023-09-10T17:46:41Z,OWNER,"In working on this I learned that `rowid` values in SQLite are way less stable than I had thought - in particular, they are often entirely rewritten on a `VACUUM`:

https://www.sqlite.org/lang_vacuum.html#how_vacuum_works

> The VACUUM command may change the [ROWIDs](https://www.sqlite.org/lang_createtable.html#rowid) of entries in any tables that do not have an explicit [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid).

So this fix wasn't as valuable as I thought. I need to move away from ever assuming that a `rowid` is a useful foreign key for anything.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,
https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708693020,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1708693020,IC_kwDOCGYnMM5l2JYc,22429695,2023-09-06T16:14:03Z,2023-11-04T00:54:25Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`347fdc8`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/347fdc865e91b8d3410f49a5c9d5b499fbb594c1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74% compared to head [(`1f14df1`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #591   +/-   ##
=======================================
  Coverage   95.74%   95.74%           
=======================================
  Files           8        8           
  Lines        2842     2842           
=======================================
  Hits         2721     2721           
  Misses        121      121           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,
https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708695907,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1708695907,IC_kwDOCGYnMM5l2KFj,9599,2023-09-06T16:15:59Z,2023-09-06T16:19:14Z,OWNER,"The test failure was while installing `numpy`, relating to importing `distutils` - maybe relevant:
- https://github.com/pypa/setuptools/issues/3661

```
25h  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
...
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_internal/utils/misc.py"", line 697, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py"", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/importlib/__init__.py"", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""<frozen importlib._bootstrap>"", line 1381, in _gcd_import
  File ""<frozen importlib._bootstrap>"", line 1354, in _find_and_load
  File ""<frozen importlib._bootstrap>"", line 1304, in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 488, in _call_with_frames_removed
  File ""<frozen importlib._bootstrap>"", line 1381, in _gcd_import
  File ""<frozen importlib._bootstrap>"", line 1354, in _find_and_load
  File ""<frozen importlib._bootstrap>"", line 1325, in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 929, in _load_unlocked
  File ""<frozen importlib._bootstrap_external>"", line 994, in exec_module
  File ""<frozen importlib._bootstrap>"", line 488, in _call_with_frames_removed
  File ""/tmp/pip-build-env-x9nyg3kd/overlay/lib/python3.12/site-packages/setuptools/__init__.py"", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,
https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1793278279,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1793278279,IC_kwDOCGYnMM5q40FH,9599,2023-11-04T00:58:03Z,2023-11-04T00:58:03Z,OWNER,I'm going to abandon this PR and ship the 3.12 testing change directly to `main`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,
https://github.com/simonw/sqlite-utils/issues/590#issuecomment-1704387161,https://api.github.com/repos/simonw/sqlite-utils/issues/590,1704387161,IC_kwDOCGYnMM5lluJZ,9599,2023-09-03T19:50:36Z,2023-09-03T19:50:36Z,OWNER,"Maybe just populate `db.memory: bool` and `db.memory_name: Optional[str]` for this, then document them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879214365,
https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704383901,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704383901,IC_kwDOCGYnMM5lltWd,9599,2023-09-03T19:34:05Z,2023-09-03T19:34:05Z,OWNER,"For that particular case I realized I'd quite like to have a mechanism for applying functions for a block of code and then de-registering them at the end - a context manager.

I played with this idea a bit:

```python
with db.register_functions(md5, md5_random):
    db.query(...)
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,
https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384111,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704384111,IC_kwDOCGYnMM5lltZv,9599,2023-09-03T19:35:03Z,2023-09-03T19:35:03Z,OWNER,"Normally in Python/`sqlite3` you de-register a function by passing `None` to it.

You can't do that with `db.register_function()` at the moment because a `fn` of `None` does something else:

https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L461-L464","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,
https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384393,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704384393,IC_kwDOCGYnMM5llteJ,9599,2023-09-03T19:36:34Z,2023-09-03T19:36:34Z,OWNER,"Here's a prototype: https://github.com/simonw/sqlite-utils/commit/62f673835c4a66f87cf6f949eaff43c8b014619b

Still needs tests and documentation (and some more thought to make sure it's doing the right thing).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,
https://github.com/simonw/sqlite-utils/issues/588#issuecomment-1694823972,https://api.github.com/repos/simonw/sqlite-utils/issues/588,1694823972,IC_kwDOCGYnMM5lBPYk,9599,2023-08-28T00:41:54Z,2023-08-28T00:41:54Z,OWNER,Tips on typing `**kwargs`: https://adamj.eu/tech/2021/05/11/python-type-hints-args-and-kwargs/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1868713944,
https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096129,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096129,IC_kwDOCGYnMM5kcIbB,9599,2023-08-19T20:03:00Z,2023-08-19T20:03:00Z,OWNER,"Simplest possible recreation of the bug:
```bash
python -c '
import sqlite_utils
db = sqlite_utils.Database(memory=True)
db.execute(""""""
CREATE TABLE ""logs"" (
   [id] INTEGER PRIMARY KEY,
   [chat_id] INTEGER REFERENCES [log]([id]),
   [reply_to_id] INTEGER
);
"""""")
db[""logs""].add_foreign_key(""reply_to_id"", ""logs"", ""id"")
'
```
That `chat_id` line is the line that causes the problem - because it is defining a reference to a table that no longer exists!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,
https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096284,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096284,IC_kwDOCGYnMM5kcIdc,9599,2023-08-19T20:03:59Z,2023-08-19T20:03:59Z,OWNER,"Although this is revealing a problem in the underlying code (that schema is invalid), it also represents a regression: `sqlite-utils 3.34` ran this just fine, but it fails on `sqlite-utils 3.35` due to the change made in:
- #577","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,
https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096381,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096381,IC_kwDOCGYnMM5kcIe9,9599,2023-08-19T20:04:32Z,2023-08-19T20:04:32Z,OWNER,I'm inclined to say this isn't a bug in `sqlite-utils` though - it's a bug in the code that calls it. So I'm not going to fix it here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,
https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683396150,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683396150,IC_kwDOCGYnMM5kVpY2,9599,2023-08-18T06:02:18Z,2023-08-18T06:06:31Z,OWNER,"More notes in here:
- https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683392873

Not all Python/SQLite installations exhibit this problem by default!

It turns out this is controlled by the `legacy_alter_table` pragma: https://sqlite.org/pragma.html#pragma_legacy_alter_table

If that PRAGMA is turned on (default in newer SQLites) then `alter table` will error if you try to rename a table that is referenced in a view.

Here's a one-liner to test if it is on or not:

```bash
python -c 'import sqlite3; print(sqlite3.connect("":memory:"").execute(""PRAGMA legacy_alter_table"").fetchall())'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,
https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683398866,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683398866,IC_kwDOCGYnMM5kVqDS,9599,2023-08-18T06:05:50Z,2023-08-18T06:06:42Z,OWNER,"Options:
- Provide a `recreate_views: bool` parameter to `table.transform()` controlling if views that might reference this table are stashed and dropped and recreated within a transaction as part of the operation. But should that be `True` or `False` by default?
- Read that `PRAGMA` and automatically do that view workaround if it's turned on
- Toggle that `PRAGMA` off for the duration of the `.transform()` operation and on again at the end. Does it only affect the current connection?
- Try the `transform()` in a transaction, detect the `""error in view""`, `""no such table""`error, if spotted then do the VIEW workaround and try again

I'm on the fence as to which of these I like the most. I'm tempted to go with the one which just drops VIEWS and recreates them all the time, because it feels simpler.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,
https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683404978,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683404978,IC_kwDOCGYnMM5kVriy,9599,2023-08-18T06:13:46Z,2023-08-18T06:13:46Z,OWNER,"I shipped the view recreating fix in `datasette-edit-schema`, so at least I can start exercising that fix and see if it has any weird issues.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683195669,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683195669,IC_kwDOCGYnMM5kU4cV,9599,2023-08-18T01:24:57Z,2023-08-18T01:24:57Z,OWNER,"Currently:
```bash
sqlite-utils transform --help
```
```
Usage: sqlite-utils transform [OPTIONS] PATH TABLE

  Transform a table beyond the capabilities of ALTER TABLE

  Example:

      sqlite-utils transform mydb.db mytable \
          --drop column1 \
          --rename column2 column_renamed

Options:
  --type <TEXT CHOICE>...   Change column type to INTEGER, TEXT, FLOAT or BLOB
  --drop TEXT               Drop this column
  --rename <TEXT TEXT>...   Rename this column to X
  -o, --column-order TEXT   Reorder columns
  --not-null TEXT           Set this column to NOT NULL
  --not-null-false TEXT     Remove NOT NULL from this column
  --pk TEXT                 Make this column the primary key
  --pk-none                 Remove primary key (convert to rowid table)
  --default <TEXT TEXT>...  Set default value for this column
  --default-none TEXT       Remove default from this column
  --drop-foreign-key TEXT   Drop foreign key constraint for this column
  --sql                     Output SQL without executing it
  --load-extension TEXT     Path to SQLite extension, with optional
                            :entrypoint
  -h, --help                Show this message and exit.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683197882,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683197882,IC_kwDOCGYnMM5kU4-6,9599,2023-08-18T01:25:53Z,2023-08-18T01:25:53Z,OWNER,"Probably most relevant here is this snippet from:
```bash
sqlite-utils create-table --help
```
```
  --default <TEXT TEXT>...  Default value that should be set for a column
  --fk <TEXT TEXT TEXT>...  Column, other table, other column to set as a
                            foreign key
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683198740,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683198740,IC_kwDOCGYnMM5kU5MU,9599,2023-08-18T01:26:47Z,2023-08-18T01:26:47Z,OWNER,"The only CLI feature that supports providing just the column name appears to be this:
```bash
sqlite-utils add-foreign-key --help
```
```
Usage: sqlite-utils add-foreign-key [OPTIONS] PATH TABLE COLUMN [OTHER_TABLE]
                                    [OTHER_COLUMN]

  Add a new foreign key constraint to an existing table

  Example:

      sqlite-utils add-foreign-key my.db books author_id authors id

  WARNING: Could corrupt your database! Back up your database file first.
```
I can drop that WARNING now since I'm not writing to `sqlite_master` any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683200128,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683200128,IC_kwDOCGYnMM5kU5iA,9599,2023-08-18T01:29:00Z,2023-08-18T01:29:00Z,OWNER,I'm not going to implement the `foreign_keys=` option that entirely replaces existing foreign keys - I'll just do a `--add-foreign-key` multi-option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683201239,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683201239,IC_kwDOCGYnMM5kU5zX,9599,2023-08-18T01:30:46Z,2023-08-18T01:30:46Z,OWNER,"Help can now look like this:
```
  --drop-foreign-key TEXT         Drop foreign key constraint for this column
  --add-foreign-key <TEXT TEXT TEXT>...
                                  Add a foreign key constraint from a column
                                  to another table with another column
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683212074,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683212074,IC_kwDOCGYnMM5kU8cq,9599,2023-08-18T01:43:54Z,2023-08-18T01:43:54Z,OWNER,"Some manual testing:
```bash
sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [places] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER,
   [city] INTEGER,
   [continent] INTEGER
);
```
```bash
sqlite-utils create-table /tmp/t.db country id integer name text
sqlite-utils create-table /tmp/t.db city id integer name text
sqlite-utils create-table /tmp/t.db continent id integer name text
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [places] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER,
   [city] INTEGER,
   [continent] INTEGER
);
CREATE TABLE [country] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [city] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [continent] (
   [id] INTEGER,
   [name] TEXT
);
```
```bash
sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [country] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [city] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [continent] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE ""places"" (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER REFERENCES [country]([id]),
   [city] INTEGER,
   [continent] INTEGER REFERENCES [continent]([id])
);
```
```bash
sqlite-utils transform /tmp/t.db places --drop-foreign-key country
sqlite-utils schema /tmp/t.db places
```
```sql
CREATE TABLE ""places"" (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER,
   [city] INTEGER,
   [continent] INTEGER REFERENCES [continent]([id])
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683217284,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683217284,IC_kwDOCGYnMM5kU9uE,9599,2023-08-18T01:50:21Z,2023-08-18T01:50:21Z,OWNER,"And a test of the `--sql` option:
```bash
sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id
sqlite-utils create-table /tmp/t.db country id integer name text
sqlite-utils create-table /tmp/t.db city id integer name text
sqlite-utils create-table /tmp/t.db continent id integer name text
sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id --sql
```
Outputs:
```sql
CREATE TABLE [places_new_6a705d2f5a13] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER REFERENCES [country]([id]),
   [city] INTEGER,
   [continent] INTEGER REFERENCES [continent]([id])
);
INSERT INTO [places_new_6a705d2f5a13] ([id], [name], [country], [city], [continent])
   SELECT [id], [name], [country], [city], [continent] FROM [places];
DROP TABLE [places];
ALTER TABLE [places_new_6a705d2f5a13] RENAME TO [places];
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112298,IC_kwDOCGYnMM5kUkFq,9599,2023-08-17T23:33:14Z,2023-08-17T23:33:14Z,OWNER,"Just one failing test left:

```
        # Soundness check foreign_keys point to existing tables
        for fk in foreign_keys:
            if fk.other_table == name and columns.get(fk.other_column):
                continue
            if not any(
                c for c in self[fk.other_table].columns if c.name == fk.other_column
            ):
>               raise AlterError(
                    ""No such column: {}.{}"".format(fk.other_table, fk.other_column)
                )
E               sqlite_utils.db.AlterError: No such column: breeds.rowid

sqlite_utils/db.py:882: AlterError
==== short test summary info ====
FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid
==== 1 failed, 378 deselected in 0.49s ====
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112857,IC_kwDOCGYnMM5kUkOZ,9599,2023-08-17T23:33:58Z,2023-08-17T23:33:58Z,OWNER,"Full test:

https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683114719,IC_kwDOCGYnMM5kUkrf,9599,2023-08-17T23:36:02Z,2023-08-17T23:36:02Z,OWNER,"Just these three lines recreate the problem:
```python
from sqlite_utils import Database

fresh_db = Database(memory=True)

fresh_db.create_table(""dogs"", {""name"": str})
fresh_db.create_table(""breeds"", {""name"": str})
fresh_db[""dogs""].add_column(""breed_id"", fk=""breeds"")
```
Traceback:
```
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2170, in add_column
    self.add_foreign_key(col_name, fk, fk_col)
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2273, in add_foreign_key
    self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1169, in add_foreign_keys
    self[table].transform(add_foreign_keys=fks)
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1728, in transform
    sqls = self.transform_sql(
           ^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1896, in transform_sql
    self.db.create_table_sql(
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 882, in create_table_sql
    raise AlterError(
sqlite_utils.db.AlterError: No such column: breeds.rowid
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683118376,IC_kwDOCGYnMM5kUlko,9599,2023-08-17T23:41:10Z,2023-08-17T23:41:10Z,OWNER,"The problem here is that the table created by this line:
```python
fresh_db.create_table(""breeds"", {""name"": str})
```
Has this schema:
```sql
CREATE TABLE [breeds] (
   [name] TEXT
);
```
SQLite creates an invisible `rowid` column for it automatically.

On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added:
```sql
(Pdb) print(fresh_db.schema)
CREATE TABLE [dogs] (
   [name] TEXT
, [breed_id] INTEGER,
   FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid])
);
CREATE TABLE [breeds] (
   [name] TEXT
);
```
But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884

Here's what the debugger reveals about this code:
```python
        for fk in foreign_keys:
            if fk.other_table == name and columns.get(fk.other_column):
                continue
            if not any(
                c for c in self[fk.other_table].columns if c.name == fk.other_column
            ):
                raise AlterError(
                    ""No such column: {}.{}"".format(fk.other_table, fk.other_column)
                )
```
```
(Pdb) fk
ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid')
(Pdb) self[fk.other_table].columns
[Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122490,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122490,IC_kwDOCGYnMM5kUmk6,22429695,2023-08-17T23:45:44Z,2023-08-18T00:46:07Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`92.85%`** and project coverage change: **`-0.07%`** :warning:
> Comparison is base [(`1dc6b5a`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/1dc6b5aa644a92d3654f7068110ed7930989ce71?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.82% compared to head [(`2915050`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.76%.
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #584      +/-   ##
==========================================
- Coverage   95.82%   95.76%   -0.07%     
==========================================
  Files           8        8              
  Lines        2829     2834       +5     
==========================================
+ Hits         2711     2714       +3     
- Misses        118      120       +2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <92.85%> (-0.15%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122767,IC_kwDOCGYnMM5kUmpP,9599,2023-08-17T23:46:09Z,2023-08-17T23:46:09Z,OWNER,"Oops, `mypy` failures:

```
sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type ""Tuple[Any, ...]"", variable has type ""Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]"")  [assignment]
sqlite_utils/db.py:1164: error: Need type annotation for ""by_table"" (hint: ""by_table: Dict[<type>, <type>] = ..."")  [var-annotated]
sqlite_utils/db.py:1169: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""transform""  [union-attr]
sqlite_utils/db.py:1813: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected <nothing>  [arg-type]
sqlite_utils/db.py:1824: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected <nothing>  [arg-type]
Found 5 errors in 1 file (checked 56 source files)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683137259,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683137259,IC_kwDOCGYnMM5kUqLr,9599,2023-08-18T00:06:59Z,2023-08-18T00:06:59Z,OWNER,"The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683138953,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683138953,IC_kwDOCGYnMM5kUqmJ,9599,2023-08-18T00:09:20Z,2023-08-18T00:09:20Z,OWNER,"Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop:
```
./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96
./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683139304,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683139304,IC_kwDOCGYnMM5kUqro,9599,2023-08-18T00:09:56Z,2023-08-18T00:09:56Z,OWNER,"Upgrading `flake8` locally replicated the error:
```
pip install -U flake8
flake8
```
```
./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96
./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683143723,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683143723,IC_kwDOCGYnMM5kUrwr,9599,2023-08-18T00:14:52Z,2023-08-18T00:14:52Z,OWNER,"Another docs update: this bit in here https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-multiple-foreign-key-constraints-at-once talks about how `.add_foreign_keys()` is a performance optimization to avoid having to run VACUUM a bunch of separate times:

> The final step in adding a new foreign key to a SQLite database is to run `VACUUM`, to ensure the new foreign key is available in future introspection queries.
> 
> `VACUUM` against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using `table.add_foreign_key(...)` these can quickly add up.
> 
> Instead, you can use `db.add_foreign_keys(...)` to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a `table`, `column`, `other_table` and `other_column`.

That doesn't apply any more - the new mechanism using `.transform()` works completely differently, so this issue around running VACUUM no longer applies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145110,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145110,IC_kwDOCGYnMM5kUsGW,9599,2023-08-18T00:16:28Z,2023-08-18T00:16:48Z,OWNER,"One last piece of documentation: need to document the new option to `table.transform()` and `table.transform_sql()`:

https://github.com/simonw/sqlite-utils/blob/0771ac61fe5c2aca74075b20b1a99b9bd4c65661/sqlite_utils/db.py#L1706-L1708

I should write tests for them too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145819,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145819,IC_kwDOCGYnMM5kUsRb,9599,2023-08-18T00:17:26Z,2023-08-18T00:17:26Z,OWNER,Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683164661,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683164661,IC_kwDOCGYnMM5kUw31,9599,2023-08-18T00:45:53Z,2023-08-18T00:45:53Z,OWNER,"More updated documentation:
- https://sqlite-utils--584.org.readthedocs.build/en/584/reference.html#sqlite_utils.db.Table.transform
- https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#python-api-transform-add-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/issues/583#issuecomment-1683110636,https://api.github.com/repos/simonw/sqlite-utils/issues/583,1683110636,IC_kwDOCGYnMM5kUjrs,9599,2023-08-17T23:31:27Z,2023-08-17T23:31:27Z,OWNER,"Spotted this while working on:
- #577 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855836914,
https://github.com/simonw/sqlite-utils/issues/581#issuecomment-1652496702,https://api.github.com/repos/simonw/sqlite-utils/issues/581,1652496702,IC_kwDOCGYnMM5ifxk-,9599,2023-07-26T21:07:45Z,2023-07-26T21:07:45Z,OWNER,Docs: https://sqlite-utils.datasette.io/en/latest/cli.html#using-the-debugger,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823160748,
https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593,https://api.github.com/repos/simonw/sqlite-utils/issues/58,710399593,MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw==,9599,2020-10-16T18:39:31Z,2020-10-16T18:39:31Z,OWNER,I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488293926,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648323482,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648323482,IC_kwDOCGYnMM5iP2ua,9599,2023-07-24T17:31:56Z,2023-07-24T17:31:56Z,OWNER,"The main blocker here is coming up with a design. The challenge is cleanly integrating it with the existing format options:

https://github.com/simonw/sqlite-utils/blob/8bee14588687b66c54c7a3dfae5de2f9cc7cac3f/docs/cli-reference.rst#L114-L132

I'd like to avoid breaking backwards compatibility.

The most obvious solution is to add a `--format x` option. This is slightly confusing as `--fmt` already exists.

Or... `--fmt` could be enhanced to handle plugins too - and the existing set of formats (from the `tabulate` integration) could be refactored to use the new hook.

If I ever do release `sqlite-utils 4` I'm tempted to move that tabulate stuff into a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648324312,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648324312,IC_kwDOCGYnMM5iP27Y,9599,2023-07-24T17:32:21Z,2023-07-24T17:32:21Z,OWNER,I'm currently leaning very slightly towards `--format geojson` - and explaining in the docs that `--format` is for formats added by plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648325682,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648325682,IC_kwDOCGYnMM5iP3Qy,9599,2023-07-24T17:33:10Z,2023-07-24T17:33:10Z,OWNER,"A related feature would be support for plugins to add new ways of _ingesting_ data - currently `sqlite-utils insert` works against JSON, newline-JSON, CSV and TSV.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648339661,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648339661,IC_kwDOCGYnMM5iP6rN,25778,2023-07-24T17:44:30Z,2023-07-24T17:44:30Z,CONTRIBUTOR,"> A related feature would be support for plugins to add new ways of ingesting data - currently sqlite-utils insert works against JSON, newline-JSON, CSV and TSV.

This is my goal, to have one plugin that handles input and output symmetrically. I'd like to be able to do something like this:

```sh
sqlite-utils insert data.db table file.geojson --format geojson
# ... explore and manipulate in Datasette
sqlite-utils query data.db ... --format geojson > output.geojson
```

This would work especially well with [datasette-query-files](https://github.com/eyeseast/datasette-query-files), since I already have the queries I need saved in standalone SQL files.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1668113177,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1668113177,IC_kwDOCGYnMM5jbWMZ,25778,2023-08-07T15:41:49Z,2023-08-07T15:41:49Z,CONTRIBUTOR,I wonder if this should be two hooks: input and output. The current `--csv` (and `--tsv`) options apply to both. Haven't looked at how it's implemented. Or maybe it's one hook that returns a format for reading and for writing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683066934,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683066934,IC_kwDOCGYnMM5kUZA2,9599,2023-08-17T22:37:18Z,2023-08-17T22:37:18Z,OWNER,"I'm certain this could work.

It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter:

https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872

Improving this code to support adding foreign keys as well would be pretty simple.

And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683068505,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683068505,IC_kwDOCGYnMM5kUZZZ,9599,2023-08-17T22:39:17Z,2023-08-17T22:39:38Z,OWNER,"This would help address these issues, among potentially many others:
- https://github.com/simonw/llm/issues/60
- https://github.com/simonw/llm/issues/116
- https://github.com/simonw/llm/issues/123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683071519,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683071519,IC_kwDOCGYnMM5kUaIf,9599,2023-08-17T22:42:28Z,2023-08-17T22:42:28Z,OWNER,"Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation:

https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149

At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`.

Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`:

https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074009,IC_kwDOCGYnMM5kUavZ,9599,2023-08-17T22:45:29Z,2023-08-17T22:47:08Z,OWNER,"Actually I think `table.transform()` might get the following optional arguments:
```python
    def transform(
        self,
        *,
        # ...
        # This one exists already:
        drop_foreign_keys: Optional[Iterable] = None,
        # These two are new. This one specifies keys to add:
        add_foreign_keys: Optional[ForeignKeysType] = None,
        # Or this one causes them all to be replaced with the new definitions:
        foreign_keys: Optional[ForeignKeysType] = None,
```
There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074546,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074546,IC_kwDOCGYnMM5kUa3y,9599,2023-08-17T22:46:18Z,2023-08-17T22:46:18Z,OWNER,"Maybe this:
```python
        drop_foreign_keys: Optional[Iterable] = None,
```
Should be this:
```python
        drop_foreign_keys: Optional[Iterable[str]] = None,
```
Because it takes a list of column names that should have their foreign keys dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074857,IC_kwDOCGYnMM5kUa8p,9599,2023-08-17T22:46:40Z,2023-08-17T22:46:40Z,OWNER,"As a reminder:
https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683076325,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683076325,IC_kwDOCGYnMM5kUbTl,9599,2023-08-17T22:48:36Z,2023-08-17T22:48:36Z,OWNER,"I'm inclined to just go with the `.transform()` method and not attempt to keep around the method that involves updating `sqlite_master` and then add code to detect if that's possible (or catch if it fails) and fall back on the other mechanism.

It would be nice to drop some code complexity, plus I don't yet have a way of running automated tests against Python + SQLite versions that exhibit the problem.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683098094,IC_kwDOCGYnMM5kUgnu,9599,2023-08-17T23:15:36Z,2023-08-17T23:15:36Z,OWNER,"An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133

Needs updating like so:
```diff
diff --git a/tests/test_extract.py b/tests/test_extract.py
index 70ad0cf..fd52534 100644
--- a/tests/test_extract.py
+++ b/tests/test_extract.py
@@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db):
     assert fresh_db[""tree""].schema == (
         'CREATE TABLE ""tree"" (\n'
         ""   [name] TEXT,\n""
-        ""   [common_name_latin_name_id] INTEGER,\n""
-        ""   FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n""
+        ""   [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n""
         "")""
     )
     assert (
```
Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in.

I don't think this should count as a breaking change release though, but it's still worth noting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1684235760,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1684235760,IC_kwDOCGYnMM5kY2Xw,9599,2023-08-18T17:43:11Z,2023-08-18T17:43:11Z,OWNER,"Here's a new plugin that brings back the `sqlite_master` modifying version, for those that can use it:

https://github.com/simonw/sqlite-utils-fast-fks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646752814,https://api.github.com/repos/simonw/sqlite-utils/issues/576,1646752814,IC_kwDOCGYnMM5iJ3Qu,9599,2023-07-23T05:43:13Z,2023-07-23T05:43:13Z,OWNER,"Using `llm` in my `sqlite-utils` local checkout:
```bash
git diff 0.2 0.3 | llm -s 'release notes as markdown bullet points'
```
> - Added a new `Table` class in the `db` module which represents a table in a SQLite database
> - The `Table` class has properties that provide information about the table, such as column names, foreign keys, and schema
> - The `Table` class also has a `count` property that returns the number of rows in the table
> - The `Table` class has a `create` method that can be used to create a new table with specified columns, primary key, and foreign keys
> - Updated the `Database` class in the `__init__` module to include the `Table` class and expose it via the `__all__` variable
> - Added new tests for creating tables and introspecting tables

```bash
git diff 0.3 0.3.1 | llm -s 'release notes as markdown bullet points'
```

> - Updated PyPI version badge to display the latest version
> - Updated Travis CI badge to link to the master branch
> - Added a badge for documentation status
> - Added a license badge
> - Added a description of the package
> - Added installation instructions
> - Added a link to the documentation
> - Added a ""Related projects"" section and linked to Datasette and csvs-to-sqlite
> - Updated the version in setup.py to 0.3.1

```bash
git diff 0.3.1 0.4 | llm -s 'release notes as markdown bullet points'
```

> - Added an `enable_fts` method to the `Table` class that enables full-text search on specified columns
> - Added a `populate_fts` method to the `Table` class that refreshes the search index for full-text search
> - Added a `search` method to the `Table` class that allows searching the table using full-text search
> - Updated the `setup.py` script to version 0.4
> - Updated the `db.py` file to include the `enable_fts`, `populate_fts`, and `search` methods
> - Added test cases for the `enable_fts`, `populate_fts`, and `search` methods in the `test_enable_fts.py` file
> - Added test cases for the `count`, `columns`, and `schema` properties in the `test_introspect.py` file
> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816997390,
https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646753678,https://api.github.com/repos/simonw/sqlite-utils/issues/576,1646753678,IC_kwDOCGYnMM5iJ3eO,9599,2023-07-23T05:49:51Z,2023-07-23T05:49:51Z,OWNER,Done - bottom of https://sqlite-utils.datasette.io/en/latest/changelog.html#id116,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816997390,
https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646687461,https://api.github.com/repos/simonw/sqlite-utils/issues/575,1646687461,IC_kwDOCGYnMM5iJnTl,9599,2023-07-22T23:01:44Z,2023-07-22T23:01:44Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/3f80a026983d3e634f05a46f2a6da162b5139dd9/sqlite_utils/db.py#L346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816919568,
https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646688288,https://api.github.com/repos/simonw/sqlite-utils/issues/575,1646688288,IC_kwDOCGYnMM5iJngg,9599,2023-07-22T23:08:22Z,2023-07-22T23:08:22Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/plugins.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816919568,