issue_comments

Trying to figure out what that extra field in table_info compared to table_xinfo is: ```

list(db.query("PRAGMA table_xinfo('t')")) [{'cid': 0, 'name': 'body', 'type': 'TEXT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 0}, {'cid': 1, 'name': 'd', 'type': 'INT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 2}] ` Presumablyhidden` 0 v.s 2 v.s. other values has meaning.

I tried it out in Jupyter and it works as advertised:

Introspection is a bit weird: there doesn't seem to be a way to introspect generated columns outside of parsing the stored SQL schema for the columns at the moment! And the .columns method doesn't return them at all:

https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L1207-L1213

Here's why:

```

db.execute("PRAGMA table_info('t')").fetchall() [(0, 'body', 'TEXT', 0, None, 0)] db.execute("PRAGMA table_xinfo('t')").fetchall() [(0, 'body', 'TEXT', 0, None, 0, 0), (1, 'd', 'INT', 0, None, 0, 2)] `` Sotable_xinfo()` is needed to get back columns including generated columns: https://www.sqlite.org/pragma.html#pragma_table_xinfo

PRAGMA schema.table_xinfo(table-name);

This pragma returns one row for each column in the named table, including hidden columns in virtual tables. The output is the same as for PRAGMA table_info except that hidden columns are shown rather than being omitted.

Just spotted this in https://www.sqlite.org/gencol.html

The only functional difference is that one cannot add new STORED columns using the ALTER TABLE ADD COLUMN command. Only VIRTUAL columns can be added using ALTER TABLE.

So to add stored columns to an existing table we would need to use the .transform() trick. Which implies that this should actually be a capability of the various .create() methods, since transform works by creating a new table with those and then copying across the old data.

Here's where .transform() calls .create_table_sql() under the hood:

https://github.com/simonw/sqlite-utils/blob/9388edf57aa15719095e3cf0952c1653cd070c9b/sqlite_utils/db.py#L1627-L1637

The Python API could be: python db[table_name].add_generated_column("field", str, "json_extract(data, '$.field')", stored=True)

Good call on adding this to create-table, especially for stored columns. Having the stored/virtual split might make this tricky to implement, but I haven't gone any farther than thinking about what the CLI looks like. I'm going to try making the SQL side work first and figure that'll tell me more about what it needs.

I'm coming round to your suggestion to have this as extra arguments to sqlite-utils add-column now, especially since you also need to pass a column type.

I'd like to come up with syntax for sqlite-utils create-table as well.

https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table

Maybe extra --generated-stored colname expression (and --generated) options would work there.

Two new parameters to .create_table() and friends:

• generated={...} - generated column definitions
• generated_stored={...} generated stored column definitions

These columns will be added at the end of the table, but you can use the column_order= parameter to apply a different order.

I like add-generated-column - feels very clear to me, and is a nice place for adding logic that checks if the DB version supports it or not and shows a useful error.