html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065597709,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065597709,IC_kwDOCGYnMM4_g7sN,9599,2022-03-11T22:32:43Z,2022-03-11T22:32:43Z,OWNER,"Trying to figure out what that extra field in `table_info` compared to `table_xinfo` is: ``` >>> list(db.query(""PRAGMA table_xinfo('t')"")) [{'cid': 0, 'name': 'body', 'type': 'TEXT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 0}, {'cid': 1, 'name': 'd', 'type': 'INT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 2}] `` Presumably `hidden` 0 v.s 2 v.s. other values has meaning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065596417,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065596417,IC_kwDOCGYnMM4_g7YB,9599,2022-03-11T22:30:15Z,2022-03-11T22:30:15Z,OWNER,"I tried it out in Jupyter and it works as advertised: Introspection is a bit weird: there doesn't seem to be a way to introspect generated columns outside of parsing the stored SQL schema for the columns at the moment! And the `.columns` method doesn't return them at all: https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L1207-L1213 Here's why: ``` >>> db.execute(""PRAGMA table_info('t')"").fetchall() [(0, 'body', 'TEXT', 0, None, 0)] >>> db.execute(""PRAGMA table_xinfo('t')"").fetchall() [(0, 'body', 'TEXT', 0, None, 0, 0), (1, 'd', 'INT', 0, None, 0, 2)] ``` So `table_xinfo()` is needed to get back columns including generated columns: https://www.sqlite.org/pragma.html#pragma_table_xinfo > **PRAGMA** *schema.***table_xinfo(***table-name***);** > > This pragma returns one row for each column in the named table, including [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables. The output is the same as for [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that hidden columns are shown rather than being omitted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065402557,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065402557,IC_kwDOCGYnMM4_gMC9,9599,2022-03-11T19:01:08Z,2022-03-11T21:42:25Z,OWNER,"Just spotted this in https://www.sqlite.org/gencol.html > The only functional difference is that one cannot add new STORED columns using the [ALTER TABLE ADD COLUMN](https://www.sqlite.org/lang_altertable.html#altertabaddcol) command. Only VIRTUAL columns can be added using ALTER TABLE. So to add stored columns to an existing table we would need to use the `.transform()` trick. Which implies that this should actually be a capability of the various `.create()` methods, since transform works by creating a new table with those and then copying across the old data. Here's where `.transform()` calls `.create_table_sql()` under the hood: https://github.com/simonw/sqlite-utils/blob/9388edf57aa15719095e3cf0952c1653cd070c9b/sqlite_utils/db.py#L1627-L1637","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065389386,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065389386,IC_kwDOCGYnMM4_gI1K,9599,2022-03-11T18:42:53Z,2022-03-11T21:40:51Z,OWNER,"The Python API could be: ```python db[table_name].add_generated_column(""field"", str, ""json_extract(data, '$.field')"", stored=True) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065458729,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065458729,IC_kwDOCGYnMM4_gZwp,9599,2022-03-11T19:58:50Z,2022-03-11T20:00:25Z,OWNER,"I'm coming round to your suggestion to have this as extra arguments to `sqlite-utils add-column` now, especially since you also need to pass a column type. I'd like to come up with syntax for `sqlite-utils create-table` as well. https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table Maybe extra `--generated-stored colname expression` (and `--generated`) options would work there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065440445,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065440445,IC_kwDOCGYnMM4_gVS9,9599,2022-03-11T19:52:15Z,2022-03-11T19:52:15Z,OWNER,"Two new parameters to `.create_table()` and friends: - `generated={...}` - generated column definitions - `generated_stored={...}` generated stored column definitions These columns will be added at the end of the table, but you can use the `column_order=` parameter to apply a different order.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065386352,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065386352,IC_kwDOCGYnMM4_gIFw,9599,2022-03-11T18:41:37Z,2022-03-11T18:41:37Z,OWNER,"I like `add-generated-column` - feels very clear to me, and is a nice place for adding logic that checks if the DB version supports it or not and shows a useful error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065384183,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065384183,IC_kwDOCGYnMM4_gHj3,9599,2022-03-11T18:40:39Z,2022-03-11T18:40:39Z,OWNER,"That fixed it: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065382145,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065382145,IC_kwDOCGYnMM4_gHEB,9599,2022-03-11T18:39:05Z,2022-03-11T18:39:05Z,OWNER,"https://sqlite-utils.datasette.io/en/3.25.1/changelog.html is currently wrong: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065381047,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065381047,IC_kwDOCGYnMM4_gGy3,9599,2022-03-11T18:38:27Z,2022-03-11T18:38:27Z,OWNER,"OK that fixed it here: https://sqlite-utils.datasette.io/en/stable/changelog.html I'm going to trigger a rebuild of `3.25.1` too: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065380286,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065380286,IC_kwDOCGYnMM4_gGm-,9599,2022-03-11T18:37:23Z,2022-03-11T18:37:23Z,OWNER,"On ReadTheDocs that triggered a new `stable` build but it didn't seem to trigger a new build of `3.25.1`: https://readthedocs.org/projects/sqlite-utils/builds/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065379528,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065379528,IC_kwDOCGYnMM4_gGbI,9599,2022-03-11T18:36:17Z,2022-03-11T18:36:17Z,OWNER,"I created a new tag and release: https://github.com/simonw/sqlite-utils/releases/tag/3.25.1 And I cancelled the publish workflow: https://github.com/simonw/sqlite-utils/actions/runs/1970415399","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065378902,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065378902,IC_kwDOCGYnMM4_gGRW,9599,2022-03-11T18:35:26Z,2022-03-11T18:35:26Z,OWNER,I deleted both the release and the tag from GitHub using the web interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065377926,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065377926,IC_kwDOCGYnMM4_gGCG,9599,2022-03-11T18:34:05Z,2022-03-11T18:34:05Z,OWNER,"Two options: - Delete and recreate the release on GitHub, triggering it to be fixed on Read The Docs (as the `stable` version) - but cancel the push to PyPI, since that platform doesn't allow package versions to be over-written and in this case since the changelog file isn't included in the PyPI package there should be no change at all - Push a 3.25.2 release. I'm going to try and do that first option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065357081,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065357081,IC_kwDOCGYnMM4_gA8Z,9599,2022-03-11T18:07:10Z,2022-03-11T18:07:10Z,OWNER,I'm really happy with this improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065345515,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065345515,IC_kwDOCGYnMM4_f-Hr,9599,2022-03-11T17:52:22Z,2022-03-11T17:52:22Z,OWNER,"Well this is a huge improvement! https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.insert I'm not crazy about the `extracts=` thing though - I wonder if there's a neat way to customize that to be less verbose?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065249883,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065249883,IC_kwDOCGYnMM4_fmxb,9599,2022-03-11T16:03:35Z,2022-03-11T16:03:35Z,OWNER,"Applying this change fixes that: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 3bc528f..2a79711 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -2293,18 +2293,18 @@ class Table(Queryable): """""" Apply conversion function ``fn`` to every value in the specified columns. - - ``columns`` - a single column or list of string column names to convert. - - ``fn`` - a callable that takes a single argument, ``value``, and returns it converted. - - ``output`` - optional string column name to write the results to (defaults to the input column). - - ``output_type`` - if the output column needs to be created, this is the type that will be used + :param columns: a single column or list of string column names to convert. + :param fn: a callable that takes a single argument, ``value``, and returns it converted. + :param output: optional string column name to write the results to (defaults to the input column). + :param output_type: if the output column needs to be created, this is the type that will be used for the new column. - - ``drop`` - boolean, should the original column be dropped once the conversion is complete? - - ``multi`` - boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a + :param drop: boolean, should the original column be dropped once the conversion is complete? + :param multi: boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a dictionary, and new columns will be created for each key of that dictionary. - - ``where`` - a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion + :param where: a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion is applied, for example ``age > ?`` or ``age > :age``. - - ``where_args`` - a list of arguments (if using ``?``) or a dictionary (if using ``:age``). - - ``show_progress`` - boolean, should a progress bar be displayed? + :param where_args: a list of arguments (if using ``?``) or a dictionary (if using ``:age``). + :param show_progress: boolean, should a progress bar be displayed? See :ref:`python_api_convert`. """""" ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065247619,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065247619,IC_kwDOCGYnMM4_fmOD,9599,2022-03-11T16:01:20Z,2022-03-11T16:01:20Z,OWNER,"Definitely an improvement! It does highlight that I'm not currently using the `:param XXX: description` syntax though, which should move my descriptions of each parameter into that generated list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065245831,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065245831,IC_kwDOCGYnMM4_flyH,9599,2022-03-11T15:59:14Z,2022-03-11T15:59:14Z,OWNER,"Hint from https://twitter.com/AdamChainz/status/1502311047612575745 > Try: > > `autodoc_typehints = 'description'` > > For a list-of-arguments format > > https://sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_typehints","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,