html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1229311612,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1229311612,IC_kwDOCGYnMM5JRc58,9599,simonw,2022-08-27T23:19:41Z,2022-08-27T23:19:41Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table - https://sqlite-utils.datasette.io/en/latest/cli.html#creating-tables ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1229206306,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1229206306,IC_kwDOCGYnMM5JRDMi,9599,simonw,2022-08-27T14:47:04Z,2022-08-27T14:47:04Z,OWNER,I could add a `--transform` option to `sqlite-utils create-table` too.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224388810,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224388810,IC_kwDOCGYnMM5I-rDK,9599,simonw,2022-08-23T17:21:16Z,2022-08-23T17:21:16Z,OWNER,Also needs comprehensive tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224386951,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224386951,IC_kwDOCGYnMM5I-qmH,9599,simonw,2022-08-23T17:20:07Z,2022-08-23T17:20:07Z,OWNER,"Example of that prototype working: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""dogs""].create({""id"": int, ""name"": str}, pk=""id"") >>> db[""dogs""].create({""id"": int, ""name"": str, ""age"": int}, pk=""id"", transform=True)
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224385575,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224385575,IC_kwDOCGYnMM5I-qQn,9599,simonw,2022-08-23T17:19:00Z,2022-08-23T17:19:00Z,OWNER,"Initial prototype: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 18a442a..03fd345 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -875,6 +875,7 @@ class Database: hash_id_columns: Optional[Iterable[str]] = None, extracts: Optional[Union[Dict[str, str], List[str]]] = None, if_not_exists: bool = False, + transform: bool = False, ) -> ""Table"": """""" Create a table with the specified name and the specified ``{column_name: type}`` columns. @@ -892,7 +893,39 @@ class Database: :param hash_id_columns: List of columns to be used when calculating the hash ID for a row :param extracts: List or dictionary of columns to be extracted during inserts, see :ref:`python_api_extracts` :param if_not_exists: Use ``CREATE TABLE IF NOT EXISTS`` - """""" + :param transform: If table already exists, transform it to fit the specified schema + """""" + # Transform table to match the new definition if table already exists: + if transform and self[name].exists(): + # First add missing columns and columns to drop + existing_columns = self[name].columns_dict + missing_columns = dict( + (col_name, col_type) + for col_name, col_type in columns.items() + if col_name not in existing_columns + ) + columns_to_drop = [ + column for column in existing_columns if column not in columns + ] + if missing_columns: + for col_name, col_type in missing_columns.items(): + self[name].add_column(col_name, col_type) + # Do we need to reset the column order? + column_order = None + if list(existing_columns) != list(columns): + column_order = list(columns) + # Only run .transform() if there is something to do + # TODO: this misses changes like pk= without also column changes + if columns_to_drop or missing_columns or column_order: + self[name].transform( + types=columns, + drop=columns_to_drop, + column_order=column_order, + not_null=not_null, + defaults=defaults, + pk=pk, + ) + return cast(Table, self[name]) sql = self.create_table_sql( name=name, columns=columns, @@ -1477,6 +1510,7 @@ class Table(Queryable): hash_id_columns: Optional[Iterable[str]] = None, extracts: Optional[Union[Dict[str, str], List[str]]] = None, if_not_exists: bool = False, + transform: bool = False, ) -> ""Table"": """""" Create a table with the specified columns. @@ -1508,6 +1542,7 @@ class Table(Queryable): hash_id_columns=hash_id_columns, extracts=extracts, if_not_exists=if_not_exists, + transform=transform, ) return self ``` Needs more thought about how things like just a change to `pk=` should work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224382336,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224382336,IC_kwDOCGYnMM5I-peA,50527,jefftriplett,2022-08-23T17:16:13Z,2022-08-23T17:16:13Z,CONTRIBUTOR,"> Should passing `alter=True` also drop any columns that aren't included in the new table structure? > > It could even spot column types that aren't correct and fix those. > > Is that consistent with the expectations set by how `alter=True` works elsewhere? I would lean towards not dropping them (or making a `drop=True` or `drop_columns=True`or `drop_missing_columns=True`) to work with existing tables easier. I do like that sqlite-utils mostly just works with existing tables but it's also nice to add to existing fields in a few cases. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224283367,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224283367,IC_kwDOCGYnMM5I-RTn,9599,simonw,2022-08-23T16:05:55Z,2022-08-23T16:05:55Z,OWNER,"... but that's what the `table.transform(...)` method does already! So maybe this is actually a `transform=True` parameter to `create()` that triggers `table.transform(...)` if necessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224280225,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224280225,IC_kwDOCGYnMM5I-Qih,9599,simonw,2022-08-23T16:03:33Z,2022-08-23T16:03:33Z,OWNER,"Maybe there should be a separate `table.alter(...)` method that does the actual work here, with `.create(..., alter=True)` as syntactic sugar for triggering that if the table exists already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224278280,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224278280,IC_kwDOCGYnMM5I-QEI,9599,simonw,2022-08-23T16:02:07Z,2022-08-23T16:02:07Z,OWNER,"Thinking about this more, I think `alter=True` is a good name for this option even if it does more than the same option on `.insert()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224272854,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224272854,IC_kwDOCGYnMM5I-OvW,9599,simonw,2022-08-23T15:58:14Z,2022-08-23T15:58:14Z,OWNER,Could call it `ensure=True` here if it works differently enough from `alter=True` that the behavior could be confusing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224271324,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224271324,IC_kwDOCGYnMM5I-OXc,9599,simonw,2022-08-23T15:56:58Z,2022-08-23T15:56:58Z,OWNER,"Should passing `alter=True` also drop any columns that aren't included in the new table structure? It could even spot column types that aren't correct and fix those. Is that consistent with the expectations set by how `alter=True` works elsewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224268273,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224268273,IC_kwDOCGYnMM5I-Nnx,9599,simonw,2022-08-23T15:54:24Z,2022-08-23T15:54:24Z,OWNER,I'm not crazy about having to pass both `alter=True` and `if_not_exists=True` - maybe `alter` should imply `if_not_exists`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224264744,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224264744,IC_kwDOCGYnMM5I-Mwo,9599,simonw,2022-08-23T15:51:29Z,2022-08-23T15:53:29Z,OWNER,"Jeff suggested `db[table].(..., alter=True)` for this. ```python db[""urls""].create( { ""url"": str, ""crawled"": bool, ""body"": str, ""headers"": dict, ""status"": int, ""status_text"": str, }, pk=""url"", defaults={""crawled"": False}, if_not_exists=True, alter=True, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns,