issue_comments

CLI design idea:

$ sqlite-utils extract my.db \
    dea_sales company_name

Here we just specify the original table and column - the new extracted table will automatically be called "company_name" and will have "id" and "value" columns, by default.

To set a custom extract table:

$ sqlite-utils extract my.db \
    dea_sales company_name \
    --table companies

And for extracting multiple columns and renaming them on the created table, maybe something like this:

$ sqlite-utils extract my.db \
    dea_sales company_name company_address \
    --table companies \
    --column company_name name \
    --column company_address address

I'll do this as part of #551

It strikes me that if scope is indeed meant to stay immutable the alternative way of solving this would be to add an outbound custom request header with the endpoint - X-Endpoint: datasette.views.table.TableView for example - and teach the Sentry plugin to optionally read that.

Huh, interesting. I'd got it into my head that scope should not be mutated under any circumstances - if that's not true and it's mutable there's all kinds of useful things we could do with it.

Asked about this on Twitter: https://twitter.com/simonw/status/1152238730259791877

Yeah that's a good call: the Datasette plugin mechanism where middleware is wrapped around the outside doesn't appear to be compatible with the Sentry mechanism of expecting that scope has been populated before it gets to their error handler.

@tomchristie is this something you've thought about?

Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command.

The operations are full table scans so calculating progress shouldn't be too difficult.

How about the Python version? That should be easier to design.

python db["dea_sales"].extract( columns=["company_name", "company_address"], to_table="companies" ) If we want to transform the extracted data (e.g. rename those columns) maybe support a transform= argument?

python db["dea_sales"].extract( columns=["company_name", "company_address"], to_table="companies", transform = lambda extracted: { "name": extracted["company_name"], "address": extracted["company_address"], } ) This would create a new "companies" table with three columns: id, name and address.

Would also be nice if there was a syntax for saying "... and use the value from this column as the primary key column in the newly created table".

So what could the interface to this look like? Especially for the CLI?

One option:

sqlite-utils extract dea_sales company_name companies name

Tricky thing here is that it's quite a large number of positional arguments:

sqlite-utils extract dea_sales company_name companies name
                     Table     column       New table New column (maybe optional?)

It would be great if this could supported multiple columns - for if a spreadsheet has e.g. a “Company Name”, “Company Address” pair of fields that always match each other and areduplicated many times.

This could be handled by creating the new table with two columns that are indexed as a unique compound key. Then you can easily get-or-create on the pairs (or triples or whatever) from the original table.

Challenge here is what does the CLI syntax look like. Something like this?

$ sqlite-utils extract dea_sales -c company_name -c company_address \
  --to companies --to-col name --to-col address

Perhaps the columns in the new table are FORCED to be the same as the old ones, hence avoiding some options? Bit restrictive… maybe they default to the same but you can customize?

$ sqlite-utils extract dea_sales -c company_name -c company_address -t companies