sqlite-utils create-view currently has a --ignore option, so adding that to sqlite-utils drop-view and sqlite-utils drop-table makes sense as well.

So: python db["my_table"].drop(ignore=True)

Other methods (db.create_view() for example) have ignore=True to mean "don't throw an error if this causes a problem", so I'm good with adding that to .drop_view().

I don't like using it as the default partly because that would be a very minor breaking API change, but mainly because I don't want to hide mistakes people make - e.g. if you mistype the name of the table you are trying to drop.

Example from the docs: ```pycon

db = sqlite_utils.Database(memory=True) db["dogs"].insert({"name": "Cleo"}) for pk, row in db["dogs"].pks_and_rows_where(): ... print(pk, row) 1 {'rowid': 1, 'name': 'Cleo'}

db["dogs_with_pk"].insert({"id": 5, "name": "Cleo"}, pk="id") for pk, row in db["dogs_with_pk"].pks_and_rows_where(): ... print(pk, row) 5 {'id': 5, 'name': 'Cleo'}

db["dogs_with_compound_pk"].insert( ... {"species": "dog", "id": 3, "name": "Cleo"}, ... pk=("species", "id") ... ) for pk, row in db["dogs_with_compound_pk"].pks_and_rows_where(): ... print(pk, row) ('dog', 3) {'species': 'dog', 'id': 3, 'name': 'Cleo'} ```

Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#listing-rows-with-their-primary-keys

WIP in a pull request.

I prototyped this and I like it: In [1]: import sqlite_utils In [2]: db = sqlite_utils.Database("/Users/simon/Dropbox/Development/datasette/fixtures.db") In [3]: list(db["compound_primary_key"].pks_and_rows_where()) Out[3]: [(('a', 'b'), {'pk1': 'a', 'pk2': 'b', 'content': 'c'})]

table.pks_and_rows_where(...) is explicit and I think less ambiguous than the other options.

I quite like pks_with_rows_where(...) - but grammatically it suggests it will return the primary keys that exist where their rows match the criteria - "pks with rows" can be interpreted as "pks for the rows that..." as opposed to "pks accompanied by rows"

Is pk_rows_where() a good name? It sounds like it returns "primary key rows" which isn't a thing. It actually returns rows along with their primary key.

Other options:

• table.rows_with_pk_where(...) - should this return (row, pk) rather than (pk, row)?
• table.rows_where_pk(...)
• table.pk_and_rows_where(...)
• table.pk_with_rows_where(...)
• table.pks_with_rows_where(...) - because rows is pluralized, so pks should be pluralized too?
• table.pks_rows_where(...)

One option: .rows_where() could grow a ensure_pk=True option which checks to see if the table is a rowid table and, if it is, includes that in the select.

Or... how about you can call .rows_where(..., pks=True) and it will yield (pk, rowdict) tuple pairs instead of just returning the sequence of dictionaries?

I'm always a little bit nervous of methods that vary their return type based on their arguments. Maybe this would be a separate method instead? python for pk, row in table.pk_rows_where(...): # ...

Here's the current implementation of .extract(): https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L1049-L1074

Tricky detail here: I create the lookup table first, based on the types of the columns that are being extracted.

I need to do this because extraction currently uses unique tuples of values, so the table has to be created in advance.

But if I'm using these new expand functions to figure out what's going to be extracted, I don't know the names of the columns and their types in advance. I'm only going to find those out during the transformation.

This may turn out to be incompatible with how .extract() works at the moment. I may need a new method, .extract_expand() perhaps? It could be simpler - work only against a single column for example.

I can still use the existing sqlite-utils extract CLI command though, with a --json flag and a rule that you can't run it against multiple columns.

Problem with calling this argument transform= is that the term "transform" already means something else in this library.

I could use convert= instead.

... but that doesn't instantly make me think of turning a value into multiple columns.

How about expand=? I've not used that term anywhere yet.

db["Reports"].extract(["Reported by"], expand={"Reported by": json.loads})

I think that works. You're expanding a single value into several columns of information.

I'm going to go with last-wins - so if multiple transform functions return the same key the last one will over-write the others.

Maybe the Python version takes an optional dictionary mapping column names to transformation functions? It could then merge all of those results together - and maybe throw an error if the same key is produced by more than one column.

python db["Reports"].extract(["Reported by"], transform={"Reported by": json.loads}) Or it could have an option for different strategies if keys collide: first wins, last wins, throw exception, add a prefix to the new column name. That feels a bit too complex for an edge-case though.

It would be OK if the CLI version only allows you to specify a single column if you are using the --json option.

For the Python version I'd like to be able to provide a transformation callback function - which can be json.loads but could also be anything else which accepts the value of the current column and returns a Python dictionary of columns and their values to use in the new table.

Likewise the sqlite-utils extract command takes one or more columns: ``` Usage: sqlite-utils extract [OPTIONS] PATH TABLE COLUMNS...

Extract one or more columns into a separate table

Options: --table TEXT Name of the other table to extract columns to --fk-column TEXT Name of the foreign key column to add to the table --rename <TEXT TEXT>... Rename this column in extracted table ```

The Python .extract() method currently starts like this: python def extract(self, columns, table=None, fk_column=None, rename=None): rename = rename or {} if isinstance(columns, str): columns = [columns] if not set(columns).issubset(self.columns_dict.keys()): raise InvalidColumns( "Invalid columns {} for table with columns {}".format( columns, list(self.columns_dict.keys()) ) ) ... Note that it takes a list of columns (and treats a string as a single item list). That's because it can be called with a list of columns and it will use them to populate another table of unique tuples of those column values.

So a new mechanism that can instead read JSON values from a single column needs to be compatible with that existing design.

I bet the bug is in here: https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L593-L602

Codecov Report

Merging #1243 (887bfd2) into main (726f781) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #1243 +/-

======================================= Coverage 91.56% 91.56%
======================================= Files 34 34
Lines 4242 4242
======================================= Hits 3884 3884
Misses 358 358
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 726f781...32652d9. Read the comment docs.