html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1224412060,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1224412060,IC_kwDOCGYnMM5I-wuc,22429695,2022-08-23T17:30:24Z,2022-08-27T23:11:44Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#468](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fc38480) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/c5f8a2eb1a81a18b52825cc649112f71fe419b12?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c5f8a2e) will **decrease** coverage by `0.12%`. > The diff coverage is `88.88%`. > :exclamation: Current head fc38480 differs from pull request most recent head 2f6a64f. Consider uploading reports for the commit 2f6a64f to get more accurate results ```diff @@ Coverage Diff @@ ## main #468 +/- ## ========================================== - Coverage 96.60% 96.47% -0.13% ========================================== Files 6 6 Lines 2590 2640 +50 ========================================== + Hits 2502 2547 +45 - Misses 88 93 +5 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/468/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.86% <66.66%> (-0.09%)` | :arrow_down: | | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/468/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.31% <90.19%> (-0.23%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436, https://github.com/simonw/datasette/pull/1792#issuecomment-1224606178,https://api.github.com/repos/simonw/datasette/issues/1792,1224606178,IC_kwDOBm6k_c5I_gHi,22429695,2022-08-23T18:49:07Z,2022-08-24T00:11:37Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1792?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1792](https://codecov.io/gh/simonw/datasette/pull/1792?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f61cfb7) into [main](https://codecov.io/gh/simonw/datasette/commit/fd1086c6867f3e3582b1eca456e4ea95f6cecf8b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fd1086c) will **increase** coverage by `0.04%`. > The diff coverage is `n/a`. ```diff @@ Coverage Diff @@ ## main #1792 +/- ## ========================================== + Coverage 91.66% 91.71% +0.04% ========================================== Files 38 38 Lines 4705 4705 ========================================== + Hits 4313 4315 +2 + Misses 392 390 -2 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1792?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1792/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.13% <0.00%> (+0.65%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348394901, https://github.com/simonw/datasette/pull/1792#issuecomment-1224869578,https://api.github.com/repos/simonw/datasette/issues/1792,1224869578,IC_kwDOBm6k_c5JAgbK,9599,2022-08-23T20:47:59Z,2022-08-23T20:47:59Z,OWNER,"Really weird: the tests are failing on Python 3.7 only for some reason, in a test that should be completely unrelated to this extension feature. https://github.com/simonw/datasette/runs/7982525580?check_suite_focus=true","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348394901, https://github.com/simonw/datasette/pull/1789#issuecomment-1224592097,https://api.github.com/repos/simonw/datasette/issues/1789,1224592097,IC_kwDOBm6k_c5I_crh,9599,2022-08-23T18:42:51Z,2022-08-23T18:42:51Z,OWNER,"Looks like it's not actually necessary to `apt-get install` anything extra in order to compile that test extension: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1344823170, https://github.com/simonw/datasette/pull/1789#issuecomment-1224569313,https://api.github.com/repos/simonw/datasette/issues/1789,1224569313,IC_kwDOBm6k_c5I_XHh,9599,2022-08-23T18:33:08Z,2022-08-23T18:33:08Z,OWNER,I'm going to merge as-is and then figure out the GitHub Actions bit separately. Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1344823170, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224388810,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224388810,IC_kwDOCGYnMM5I-rDK,9599,2022-08-23T17:21:16Z,2022-08-23T17:21:16Z,OWNER,Also needs comprehensive tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224386951,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224386951,IC_kwDOCGYnMM5I-qmH,9599,2022-08-23T17:20:07Z,2022-08-23T17:20:07Z,OWNER,"Example of that prototype working: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""dogs""].create({""id"": int, ""name"": str}, pk=""id"") >>> db[""dogs""].create({""id"": int, ""name"": str, ""age"": int}, pk=""id"", transform=True)
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224385575,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224385575,IC_kwDOCGYnMM5I-qQn,9599,2022-08-23T17:19:00Z,2022-08-23T17:19:00Z,OWNER,"Initial prototype: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 18a442a..03fd345 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -875,6 +875,7 @@ class Database: hash_id_columns: Optional[Iterable[str]] = None, extracts: Optional[Union[Dict[str, str], List[str]]] = None, if_not_exists: bool = False, + transform: bool = False, ) -> ""Table"": """""" Create a table with the specified name and the specified ``{column_name: type}`` columns. @@ -892,7 +893,39 @@ class Database: :param hash_id_columns: List of columns to be used when calculating the hash ID for a row :param extracts: List or dictionary of columns to be extracted during inserts, see :ref:`python_api_extracts` :param if_not_exists: Use ``CREATE TABLE IF NOT EXISTS`` - """""" + :param transform: If table already exists, transform it to fit the specified schema + """""" + # Transform table to match the new definition if table already exists: + if transform and self[name].exists(): + # First add missing columns and columns to drop + existing_columns = self[name].columns_dict + missing_columns = dict( + (col_name, col_type) + for col_name, col_type in columns.items() + if col_name not in existing_columns + ) + columns_to_drop = [ + column for column in existing_columns if column not in columns + ] + if missing_columns: + for col_name, col_type in missing_columns.items(): + self[name].add_column(col_name, col_type) + # Do we need to reset the column order? + column_order = None + if list(existing_columns) != list(columns): + column_order = list(columns) + # Only run .transform() if there is something to do + # TODO: this misses changes like pk= without also column changes + if columns_to_drop or missing_columns or column_order: + self[name].transform( + types=columns, + drop=columns_to_drop, + column_order=column_order, + not_null=not_null, + defaults=defaults, + pk=pk, + ) + return cast(Table, self[name]) sql = self.create_table_sql( name=name, columns=columns, @@ -1477,6 +1510,7 @@ class Table(Queryable): hash_id_columns: Optional[Iterable[str]] = None, extracts: Optional[Union[Dict[str, str], List[str]]] = None, if_not_exists: bool = False, + transform: bool = False, ) -> ""Table"": """""" Create a table with the specified columns. @@ -1508,6 +1542,7 @@ class Table(Queryable): hash_id_columns=hash_id_columns, extracts=extracts, if_not_exists=if_not_exists, + transform=transform, ) return self ``` Needs more thought about how things like just a change to `pk=` should work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224382336,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224382336,IC_kwDOCGYnMM5I-peA,50527,2022-08-23T17:16:13Z,2022-08-23T17:16:13Z,CONTRIBUTOR,"> Should passing `alter=True` also drop any columns that aren't included in the new table structure? > > It could even spot column types that aren't correct and fix those. > > Is that consistent with the expectations set by how `alter=True` works elsewhere? I would lean towards not dropping them (or making a `drop=True` or `drop_columns=True`or `drop_missing_columns=True`) to work with existing tables easier. I do like that sqlite-utils mostly just works with existing tables but it's also nice to add to existing fields in a few cases. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224283367,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224283367,IC_kwDOCGYnMM5I-RTn,9599,2022-08-23T16:05:55Z,2022-08-23T16:05:55Z,OWNER,"... but that's what the `table.transform(...)` method does already! So maybe this is actually a `transform=True` parameter to `create()` that triggers `table.transform(...)` if necessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224280225,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224280225,IC_kwDOCGYnMM5I-Qih,9599,2022-08-23T16:03:33Z,2022-08-23T16:03:33Z,OWNER,"Maybe there should be a separate `table.alter(...)` method that does the actual work here, with `.create(..., alter=True)` as syntactic sugar for triggering that if the table exists already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224278280,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224278280,IC_kwDOCGYnMM5I-QEI,9599,2022-08-23T16:02:07Z,2022-08-23T16:02:07Z,OWNER,"Thinking about this more, I think `alter=True` is a good name for this option even if it does more than the same option on `.insert()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224272854,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224272854,IC_kwDOCGYnMM5I-OvW,9599,2022-08-23T15:58:14Z,2022-08-23T15:58:14Z,OWNER,Could call it `ensure=True` here if it works differently enough from `alter=True` that the behavior could be confusing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224271324,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224271324,IC_kwDOCGYnMM5I-OXc,9599,2022-08-23T15:56:58Z,2022-08-23T15:56:58Z,OWNER,"Should passing `alter=True` also drop any columns that aren't included in the new table structure? It could even spot column types that aren't correct and fix those. Is that consistent with the expectations set by how `alter=True` works elsewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224268273,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224268273,IC_kwDOCGYnMM5I-Nnx,9599,2022-08-23T15:54:24Z,2022-08-23T15:54:24Z,OWNER,I'm not crazy about having to pass both `alter=True` and `if_not_exists=True` - maybe `alter` should imply `if_not_exists`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224264744,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224264744,IC_kwDOCGYnMM5I-Mwo,9599,2022-08-23T15:51:29Z,2022-08-23T15:53:29Z,OWNER,"Jeff suggested `db[table].(..., alter=True)` for this. ```python db[""urls""].create( { ""url"": str, ""crawled"": bool, ""body"": str, ""headers"": dict, ""status"": int, ""status_text"": str, }, pk=""url"", defaults={""crawled"": False}, if_not_exists=True, alter=True, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997, https://github.com/simonw/datasette/issues/1791#issuecomment-1224080105,https://api.github.com/repos/simonw/datasette/issues/1791,1224080105,IC_kwDOBm6k_c5I9frp,1780782,2022-08-23T13:29:51Z,2022-08-23T13:29:51Z,NONE,"For now I've installed datasette via homebrew. I then converted my CSV to SQLite via sqlite-utils as per the article [Cleaning data with sqlite](https://datasette.io/tutorials/clean-data). Now I can use my `metadata.json` file as follows: `datasette mydb.db -m mymetadata.json` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1347717749, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1223378004,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1223378004,IC_kwDOCGYnMM5I60RU,1558033,2022-08-23T00:44:11Z,2022-08-23T00:44:11Z,NONE,"This bug affects me as well. Env: ``` Python 3.8.12 sqlite-utils, version 3.28 sqlite3 3.32.3 MacOS Big Sur 11.6.7 Intel ``` Similar to @mdrovdahl, I was able to work around this bug by piping the SQL string constructed in `add_foreign_keys` to the `sqlite3` command itself. Specifically, if you're trying to patch this yourself, replace [lines 1026-1039 of db.py in your site packages](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L1026-L1039) with something similar to the following: ``` print(""PRAGMA writable_schema = 1;"") for table_name, new_sql in table_sql.items(): print(""UPDATE sqlite_master SET sql = '{}' WHERE name = '{}';"".format( new_sql, table_name) ) print(""PRAGMA writable_schema = 0;"") print(""VACUUM;"") ``` Then from your terminal: `db-to-sqlite """" your.db --all > output.sql && sqlite3 your.db < output.sql` If you want to run this with `-p`, you'll have to actually open a file in code to write to instead of redirecting the output.","{""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 3, ""rocket"": 0, ""eyes"": 0}",810618495, https://github.com/simonw/datasette/pull/1789#issuecomment-1223347322,https://api.github.com/repos/simonw/datasette/issues/1789,1223347322,IC_kwDOBm6k_c5I6sx6,15178711,2022-08-23T00:03:20Z,2022-08-23T00:03:20Z,CONTRIBUTOR,"@simonw to build the extension on ubuntu, you can run: ``` apt-get update && apt-get install libsqlite3-dev gcc gcc ext.c -fPIC -shared -o ext.so ``` I'm not the best with Actions, but if you set the cache key to `ext.c`, run those two commands to download dependencies + compile to `ext.so`, then the unit test should pick it up and run it correctly. Let me know if you want me to update the PR with that added","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1344823170,