html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065597709,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065597709,IC_kwDOCGYnMM4_g7sN,9599,2022-03-11T22:32:43Z,2022-03-11T22:32:43Z,OWNER,"Trying to figure out what that extra field in `table_info` compared to `table_xinfo` is:
```
>>> list(db.query(""PRAGMA table_xinfo('t')""))
[{'cid': 0,
'name': 'body',
'type': 'TEXT',
'notnull': 0,
'dflt_value': None,
'pk': 0,
'hidden': 0},
{'cid': 1,
'name': 'd',
'type': 'INT',
'notnull': 0,
'dflt_value': None,
'pk': 0,
'hidden': 2}]
``
Presumably `hidden` 0 v.s 2 v.s. other values has meaning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,
https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065596417,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065596417,IC_kwDOCGYnMM4_g7YB,9599,2022-03-11T22:30:15Z,2022-03-11T22:30:15Z,OWNER,"I tried it out in Jupyter and it works as advertised:
Introspection is a bit weird: there doesn't seem to be a way to introspect generated columns outside of parsing the stored SQL schema for the columns at the moment! And the `.columns` method doesn't return them at all:
https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L1207-L1213
Here's why:
```
>>> db.execute(""PRAGMA table_info('t')"").fetchall()
[(0, 'body', 'TEXT', 0, None, 0)]
>>> db.execute(""PRAGMA table_xinfo('t')"").fetchall()
[(0, 'body', 'TEXT', 0, None, 0, 0), (1, 'd', 'INT', 0, None, 0, 2)]
```
So `table_xinfo()` is needed to get back columns including generated columns: https://www.sqlite.org/pragma.html#pragma_table_xinfo
> **PRAGMA** *schema.***table_xinfo(***table-name***);**
>
> This pragma returns one row for each column in the named table, including [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables. The output is the same as for [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that hidden columns are shown rather than being omitted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,
https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065402557,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065402557,IC_kwDOCGYnMM4_gMC9,9599,2022-03-11T19:01:08Z,2022-03-11T21:42:25Z,OWNER,"Just spotted this in https://www.sqlite.org/gencol.html
> The only functional difference is that one cannot add new STORED columns using the [ALTER TABLE ADD COLUMN](https://www.sqlite.org/lang_altertable.html#altertabaddcol) command. Only VIRTUAL columns can be added using ALTER TABLE.
So to add stored columns to an existing table we would need to use the `.transform()` trick. Which implies that this should actually be a capability of the various `.create()` methods, since transform works by creating a new table with those and then copying across the old data.
Here's where `.transform()` calls `.create_table_sql()` under the hood:
https://github.com/simonw/sqlite-utils/blob/9388edf57aa15719095e3cf0952c1653cd070c9b/sqlite_utils/db.py#L1627-L1637","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,
https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065389386,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065389386,IC_kwDOCGYnMM4_gI1K,9599,2022-03-11T18:42:53Z,2022-03-11T21:40:51Z,OWNER,"The Python API could be:
```python
db[table_name].add_generated_column(""field"", str, ""json_extract(data, '$.field')"", stored=True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,
https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065458729,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065458729,IC_kwDOCGYnMM4_gZwp,9599,2022-03-11T19:58:50Z,2022-03-11T20:00:25Z,OWNER,"I'm coming round to your suggestion to have this as extra arguments to `sqlite-utils add-column` now, especially since you also need to pass a column type.
I'd like to come up with syntax for `sqlite-utils create-table` as well.
https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table
Maybe extra `--generated-stored colname expression` (and `--generated`) options would work there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,
https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065440445,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065440445,IC_kwDOCGYnMM4_gVS9,9599,2022-03-11T19:52:15Z,2022-03-11T19:52:15Z,OWNER,"Two new parameters to `.create_table()` and friends:
- `generated={...}` - generated column definitions
- `generated_stored={...}` generated stored column definitions
These columns will be added at the end of the table, but you can use the `column_order=` parameter to apply a different order.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1160034488,
https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065386352,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065386352,IC_kwDOCGYnMM4_gIFw,9599,2022-03-11T18:41:37Z,2022-03-11T18:41:37Z,OWNER,"I like `add-generated-column` - feels very clear to me, and is a nice place for adding logic that checks if the DB version supports it or not and shows a useful error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,
https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065384183,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065384183,IC_kwDOCGYnMM4_gHj3,9599,2022-03-11T18:40:39Z,2022-03-11T18:40:39Z,OWNER,"That fixed it:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,
https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065382145,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065382145,IC_kwDOCGYnMM4_gHEB,9599,2022-03-11T18:39:05Z,2022-03-11T18:39:05Z,OWNER,"https://sqlite-utils.datasette.io/en/3.25.1/changelog.html is currently wrong:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,
https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065381047,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065381047,IC_kwDOCGYnMM4_gGy3,9599,2022-03-11T18:38:27Z,2022-03-11T18:38:27Z,OWNER,"OK that fixed it here: https://sqlite-utils.datasette.io/en/stable/changelog.html
I'm going to trigger a rebuild of `3.25.1` too:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,
https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065380286,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065380286,IC_kwDOCGYnMM4_gGm-,9599,2022-03-11T18:37:23Z,2022-03-11T18:37:23Z,OWNER,"On ReadTheDocs that triggered a new `stable` build but it didn't seem to trigger a new build of `3.25.1`: https://readthedocs.org/projects/sqlite-utils/builds/
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,
https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065379528,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065379528,IC_kwDOCGYnMM4_gGbI,9599,2022-03-11T18:36:17Z,2022-03-11T18:36:17Z,OWNER,"I created a new tag and release: https://github.com/simonw/sqlite-utils/releases/tag/3.25.1
And I cancelled the publish workflow: https://github.com/simonw/sqlite-utils/actions/runs/1970415399","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,
https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065378902,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065378902,IC_kwDOCGYnMM4_gGRW,9599,2022-03-11T18:35:26Z,2022-03-11T18:35:26Z,OWNER,I deleted both the release and the tag from GitHub using the web interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,
https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065377926,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065377926,IC_kwDOCGYnMM4_gGCG,9599,2022-03-11T18:34:05Z,2022-03-11T18:34:05Z,OWNER,"Two options:
- Delete and recreate the release on GitHub, triggering it to be fixed on Read The Docs (as the `stable` version) - but cancel the push to PyPI, since that platform doesn't allow package versions to be over-written and in this case since the changelog file isn't included in the PyPI package there should be no change at all
- Push a 3.25.2 release.
I'm going to try and do that first option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,
https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065357081,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065357081,IC_kwDOCGYnMM4_gA8Z,9599,2022-03-11T18:07:10Z,2022-03-11T18:07:10Z,OWNER,I'm really happy with this improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,
https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065345515,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065345515,IC_kwDOCGYnMM4_f-Hr,9599,2022-03-11T17:52:22Z,2022-03-11T17:52:22Z,OWNER,"Well this is a huge improvement!
https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.insert
I'm not crazy about the `extracts=` thing though - I wonder if there's a neat way to customize that to be less verbose?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,
https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065249883,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065249883,IC_kwDOCGYnMM4_fmxb,9599,2022-03-11T16:03:35Z,2022-03-11T16:03:35Z,OWNER,"Applying this change fixes that:
```diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index 3bc528f..2a79711 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -2293,18 +2293,18 @@ class Table(Queryable):
""""""
Apply conversion function ``fn`` to every value in the specified columns.
- - ``columns`` - a single column or list of string column names to convert.
- - ``fn`` - a callable that takes a single argument, ``value``, and returns it converted.
- - ``output`` - optional string column name to write the results to (defaults to the input column).
- - ``output_type`` - if the output column needs to be created, this is the type that will be used
+ :param columns: a single column or list of string column names to convert.
+ :param fn: a callable that takes a single argument, ``value``, and returns it converted.
+ :param output: optional string column name to write the results to (defaults to the input column).
+ :param output_type: if the output column needs to be created, this is the type that will be used
for the new column.
- - ``drop`` - boolean, should the original column be dropped once the conversion is complete?
- - ``multi`` - boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a
+ :param drop: boolean, should the original column be dropped once the conversion is complete?
+ :param multi: boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a
dictionary, and new columns will be created for each key of that dictionary.
- - ``where`` - a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion
+ :param where: a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion
is applied, for example ``age > ?`` or ``age > :age``.
- - ``where_args`` - a list of arguments (if using ``?``) or a dictionary (if using ``:age``).
- - ``show_progress`` - boolean, should a progress bar be displayed?
+ :param where_args: a list of arguments (if using ``?``) or a dictionary (if using ``:age``).
+ :param show_progress: boolean, should a progress bar be displayed?
See :ref:`python_api_convert`.
""""""
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,
https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065247619,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065247619,IC_kwDOCGYnMM4_fmOD,9599,2022-03-11T16:01:20Z,2022-03-11T16:01:20Z,OWNER,"Definitely an improvement!
It does highlight that I'm not currently using the `:param XXX: description` syntax though, which should move my descriptions of each parameter into that generated list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,
https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065245831,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065245831,IC_kwDOCGYnMM4_flyH,9599,2022-03-11T15:59:14Z,2022-03-11T15:59:14Z,OWNER,"Hint from https://twitter.com/AdamChainz/status/1502311047612575745
> Try:
>
> `autodoc_typehints = 'description'`
>
> For a list-of-arguments format
>
> https://sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_typehints","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,