issue_comments

Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command.

The operations are full table scans so calculating progress shouldn't be too difficult.

How about the Python version? That should be easier to design.

python db["dea_sales"].extract( columns=["company_name", "company_address"], to_table="companies" ) If we want to transform the extracted data (e.g. rename those columns) maybe support a transform= argument?

python db["dea_sales"].extract( columns=["company_name", "company_address"], to_table="companies", transform = lambda extracted: { "name": extracted["company_name"], "address": extracted["company_address"], } ) This would create a new "companies" table with three columns: id, name and address.

Would also be nice if there was a syntax for saying "... and use the value from this column as the primary key column in the newly created table".

So what could the interface to this look like? Especially for the CLI?

One option:

sqlite-utils extract dea_sales company_name companies name

Tricky thing here is that it's quite a large number of positional arguments:

sqlite-utils extract dea_sales company_name companies name
                     Table     column       New table New column (maybe optional?)

It would be great if this could supported multiple columns - for if a spreadsheet has e.g. a “Company Name”, “Company Address” pair of fields that always match each other and areduplicated many times.

This could be handled by creating the new table with two columns that are indexed as a unique compound key. Then you can easily get-or-create on the pairs (or triples or whatever) from the original table.

Challenge here is what does the CLI syntax look like. Something like this?

$ sqlite-utils extract dea_sales -c company_name -c company_address \
  --to companies --to-col name --to-col address

Perhaps the columns in the new table are FORCED to be the same as the old ones, hence avoiding some options? Bit restrictive… maybe they default to the same but you can customize?

$ sqlite-utils extract dea_sales -c company_name -c company_address -t companies