issue_comments

Trying to figure out what that extra field in table_info compared to table_xinfo is: ```

list(db.query("PRAGMA table_xinfo('t')")) [{'cid': 0, 'name': 'body', 'type': 'TEXT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 0}, {'cid': 1, 'name': 'd', 'type': 'INT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 2}] ` Presumablyhidden` 0 v.s 2 v.s. other values has meaning.

I tried it out in Jupyter and it works as advertised:

Introspection is a bit weird: there doesn't seem to be a way to introspect generated columns outside of parsing the stored SQL schema for the columns at the moment! And the .columns method doesn't return them at all:

https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L1207-L1213

Here's why:

```

db.execute("PRAGMA table_info('t')").fetchall() [(0, 'body', 'TEXT', 0, None, 0)] db.execute("PRAGMA table_xinfo('t')").fetchall() [(0, 'body', 'TEXT', 0, None, 0, 0), (1, 'd', 'INT', 0, None, 0, 2)] `` Sotable_xinfo()` is needed to get back columns including generated columns: https://www.sqlite.org/pragma.html#pragma_table_xinfo

PRAGMA schema.table_xinfo(table-name);

This pragma returns one row for each column in the named table, including hidden columns in virtual tables. The output is the same as for PRAGMA table_info except that hidden columns are shown rather than being omitted.

Just spotted this in https://www.sqlite.org/gencol.html

The only functional difference is that one cannot add new STORED columns using the ALTER TABLE ADD COLUMN command. Only VIRTUAL columns can be added using ALTER TABLE.

So to add stored columns to an existing table we would need to use the .transform() trick. Which implies that this should actually be a capability of the various .create() methods, since transform works by creating a new table with those and then copying across the old data.

Here's where .transform() calls .create_table_sql() under the hood:

https://github.com/simonw/sqlite-utils/blob/9388edf57aa15719095e3cf0952c1653cd070c9b/sqlite_utils/db.py#L1627-L1637

The Python API could be: python db[table_name].add_generated_column("field", str, "json_extract(data, '$.field')", stored=True)

Good call on adding this to create-table, especially for stored columns. Having the stored/virtual split might make this tricky to implement, but I haven't gone any farther than thinking about what the CLI looks like. I'm going to try making the SQL side work first and figure that'll tell me more about what it needs.

I'm coming round to your suggestion to have this as extra arguments to sqlite-utils add-column now, especially since you also need to pass a column type.

I'd like to come up with syntax for sqlite-utils create-table as well.

https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table

Maybe extra --generated-stored colname expression (and --generated) options would work there.

Two new parameters to .create_table() and friends:

• generated={...} - generated column definitions
• generated_stored={...} generated stored column definitions

These columns will be added at the end of the table, but you can use the column_order= parameter to apply a different order.

I like add-generated-column - feels very clear to me, and is a nice place for adding logic that checks if the DB version supports it or not and shows a useful error.

That fixed it:

https://sqlite-utils.datasette.io/en/3.25.1/changelog.html is currently wrong:

OK that fixed it here: https://sqlite-utils.datasette.io/en/stable/changelog.html

I'm going to trigger a rebuild of 3.25.1 too:

On ReadTheDocs that triggered a new stable build but it didn't seem to trigger a new build of 3.25.1: https://readthedocs.org/projects/sqlite-utils/builds/

I created a new tag and release: https://github.com/simonw/sqlite-utils/releases/tag/3.25.1

And I cancelled the publish workflow: https://github.com/simonw/sqlite-utils/actions/runs/1970415399

I deleted both the release and the tag from GitHub using the web interface.

Two options:

• Delete and recreate the release on GitHub, triggering it to be fixed on Read The Docs (as the stable version) - but cancel the push to PyPI, since that platform doesn't allow package versions to be over-written and in this case since the changelog file isn't included in the PyPI package there should be no change at all
• Push a 3.25.2 release.

I'm going to try and do that first option.

I'm really happy with this improvement.

Well this is a huge improvement!

https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.insert

I'm not crazy about the extracts=<sqlite_utils.db.Default object> thing though - I wonder if there's a neat way to customize that to be less verbose?

I noticed the image was large when using fly. Is it possible to use a -slim base?

Applying this change fixes that: `diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 3bc528f..2a79711 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -2293,18 +2293,18 @@ class Table(Queryable): """ Apply conversion functionfn`` to every value in the specified columns.

• • columns - a single column or list of string column names to convert.
• • fn - a callable that takes a single argument, value, and returns it converted.
• • output - optional string column name to write the results to (defaults to the input column).
• • output_type - if the output column needs to be created, this is the type that will be used
• :param columns: a single column or list of string column names to convert.
• :param fn: a callable that takes a single argument, value, and returns it converted.
• :param output: optional string column name to write the results to (defaults to the input column).
• :param output_type: if the output column needs to be created, this is the type that will be used for the new column.
• • drop - boolean, should the original column be dropped once the conversion is complete?
• • multi - boolean, if True the return value of fn(value) will be expected to be a
• :param drop: boolean, should the original column be dropped once the conversion is complete?
• :param multi: boolean, if True the return value of fn(value) will be expected to be a dictionary, and new columns will be created for each key of that dictionary.
• • where - a SQL fragment to use as a WHERE clause to limit the rows to which the conversion
• :param where: a SQL fragment to use as a WHERE clause to limit the rows to which the conversion is applied, for example age > ? or age > :age.
• • where_args - a list of arguments (if using ?) or a dictionary (if using :age).
• • show_progress - boolean, should a progress bar be displayed?
• :param where_args: a list of arguments (if using ?) or a dictionary (if using :age).
• :param show_progress: boolean, should a progress bar be displayed?

   See :ref:`python_api_convert`.
   """
  

  ```

Definitely an improvement!

It does highlight that I'm not currently using the :param XXX: description syntax though, which should move my descriptions of each parameter into that generated list.

Hint from https://twitter.com/AdamChainz/status/1502311047612575745

Try:

autodoc_typehints = 'description'

For a list-of-arguments format

https://sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_typehints