Documentation for sqlite-utils extract: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table

Trying this demo again: wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name It worked!

The documentation for the .extract() method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table

I tried out the prototype of the CLI on the Global Power Plants data: wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long This threw an error because rowid columns are not yet supported. I fixed that like so: sqlite-utils transform global.db power_plants --rename rowid id sqlite-utils extract global.db power_plants country country_long That worked! But it didn't play great with Datasette, because the resulting extracted table had columns country and country_long and neither of those are called name or value or title.

Based on this I need to add rowid table support AND I need to implement the proposed rename= argument for renaming columns on their way into the new table.

The command-line version of this needs to accept a table and one or more columns, then a --table and --fk-column option.

Here's how I'm generating the examples for the documentation: ``` In [2]: import sqlite_utils

In [3]: db = sqlite_utils.Database(memory=True)

In [4]: db["Trees"].insert({"id": 1, "TreeAddress": "52 Vine St", "CommonName": ...: "Palm", "LatinName": "foo"}, pk="id") Out[4]: <Table Trees (id, TreeAddress, CommonName, LatinName)>

In [5]: db["Trees"].extract(["CommonName", "LatinName"], table="Species", fk_col ...: umn="species_id")

In [6]: print(db["Trees"].schema) CREATE TABLE "Trees" ( [id] INTEGER PRIMARY KEY, [TreeAddress] TEXT, [species_id] INTEGER, FOREIGN KEY(species_id) REFERENCES Species(id) )

In [7]: print(db["Species"].schema) CREATE TABLE [Species] ( [id] INTEGER PRIMARY KEY, [CommonName] TEXT, [LatinName] TEXT ) ```

Need to make sure this works correctly for rowid tables.

What to do if the table already exists? The .lookup() function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.

So .extract() probably takes a batch_size= argument too, which defaults to maybe 1000.

table.extract() can take an optional progress= argument which is a callback which will be used to report progress - called after each batch with (num_done, total). It will get called with (0, total) once at the start to allow progress bars to be initialized. The command-line progress bar will use this.

And if you want to rename some of the columns in the new table: python db["trees"].extract(["common_name", "latin_name"], table="species", rename={"common_name": "name"})

In Python it looks like this: ```python

Simple case - species column species_id pointing to species table

db["trees"].extract("species")

Setting a custom table

db["trees"].extract("species", table="Species")

Custom foreign key column on trees

db["trees"].extract("species", fk_column="species")

Extracting multiple columns

db["trees"].extract(["common_name", "latin_name"])

(this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id)

Or with explicit table (fk_column here defaults to species_id because of the table name)

db["trees"].extract(["common_name", "latin_name"], table="species") ```

I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the .lookup() mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID.

That integer ID gets written back into the first of the columns that are being transformed. A .transform() call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.

I think the new foreign key column is called company_name_id by default in this example but can be customized by passing --fk-column=xxx

Thinking more about this one: $ sqlite-utils extract my.db \ dea_sales company_name company_address \ --table companies The goal here is to pull the company name and address pair out into a separate table.

Some questions: - should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this. - what should the foreign key column that gets added to the companies table be called?

Confirmed in local dev: % datasette fixtures.db --inspect-file inspect.json Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette", line 11, in <module> load_entry_point('datasette', 'console_scripts', 'datasette')() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 406, in serve inspect_data = json.load(open(inspect_file)) TypeError: 'bool' object is not callable

D'oh because I have a new variable called open.

I don't think a standard "pass these extra arguments to the publish tool" mechanism will work because there's no guarantee that a publisher uses a CLI tool - or if it does, it might make several calls to different CLI tools. The Cloud Run one runs a couple of commands, as illustrated by this test:

https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/tests/test_publish_cloudrun.py#L63-L73

Adding a --tar option for datasette publish heroku is a good fix for this though.

Am I going to rewrite ALL of my tests to use this instead? It would clean up a lot of test code, at the cost of quite a bit of work.

It would make for much neater plugin tests too, and neater testing documentation: https://docs.datasette.io/en/stable/testing_plugins.html

```python class DatasetteClient: def init(self, ds): self._client = httpx.AsyncClient(app=ds.app())

def _fix(self, path):
    if path.startswith("/"):
        path = "http://localhost{}".format(path)
    return path

async def get(self, path, **kwargs):
    return await self._client.get(self._fix(path), **kwargs)

async def options(self, path, **kwargs):
    return await self._client.options(self._fix(path), **kwargs)

async def head(self, path, **kwargs):
    return await self._client.head(self._fix(path), **kwargs)

async def post(self, path, **kwargs):
    return await self._client.post(self._fix(path), **kwargs)

async def put(self, path, **kwargs):
    return await self._client.put(self._fix(path), **kwargs)

async def patch(self, path, **kwargs):
    return await self._client.patch(self._fix(path), **kwargs)

async def delete(self, path, **kwargs):
    return await self._client.delete(self._fix(path), **kwargs)

```

I may as well implement all of the HTTP methods supported by the httpx client:

• get
• options
• head
• post
• put
• patch
• delete

Even smaller DatasetteClient implementation: ```python class DatasetteClient: def init(self, ds): self._client = httpx.AsyncClient(app=ds.app())

def _fix(self, path):
    if path.startswith("/"):
        path = "http://localhost{}".format(path)
    return path

async def get(self, path, **kwargs):
    return await self._client.get(self._fix(path), **kwargs)

async def post(self, path, **kwargs):
    return await self._client.post(self._fix(path), **kwargs)

async def options(self, path, **kwargs):
    return await self._client.options(self._fix(path), **kwargs)

```

How important is it to use httpx.AsyncClient with a context manager?

https://www.python-httpx.org/async/#opening-and-closing-clients says:

Alternatively, use await client.aclose() if you want to close a client explicitly:

client = httpx.AsyncClient() ... await client.aclose() The .aclose() method has a comment saying "Close transport and proxies" - I'm not using proxies, so the relevant implementation seems to be a call to await self._transport.aclose() in https://github.com/encode/httpx/blob/f932af9172d15a803ad40061a4c2c0cd891645cf/httpx/_client.py#L1741-L1751

The transport I am using is a class called ASGITransport in https://github.com/encode/httpx/blob/master/httpx/_transports/asgi.py

The aclose() method on that class does nothing. So it looks like I can instantiate a client without bothering with the async with httpx.AsyncClient bit.

This adds httpx as a dependency - I think I'm OK with that. I use it for testing in all of my plugins anyway.

I put together a minimal prototype of this and it feels pretty good: ```diff diff --git a/datasette/app.py b/datasette/app.py index 20aae7d..fb3bdad 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -4,6 +4,7 @@ import collections import datetime import glob import hashlib +import httpx import inspect import itertools from itsdangerous import BadSignature @@ -312,6 +313,7 @@ class Datasette: self._register_renderers() self._permission_checks = collections.deque(maxlen=200) self._root_token = secrets.token_hex(32) + self.client = DatasetteClient(self)

 async def invoke_startup(self):
     for hook in pm.hook.startup(datasette=self):

@@ -1209,3 +1211,25 @@ def route_pattern_from_filepath(filepath):

class NotFoundExplicit(NotFound): pass + + +class DatasetteClient: + def init(self, ds): + self.app = ds.app() + + def _fix(self, path): + if path.startswith("/"): + path = "http://localhost{}".format(path) + return path + + async def get(self, path, kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.get(self._fix(path), kwargs) + + async def post(self, path, kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.post(self._fix(path), kwargs) + + async def options(self, path, kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.options(self._fix(path), kwargs) Used like this in `ipython`: In [1]: from datasette.app import Datasette

In [2]: ds = Datasette(["fixtures.db"])

In [3]: (await ds.client.get("/-/config.json")).json() Out[3]: {'default_page_size': 100, 'max_returned_rows': 1000, 'num_sql_threads': 3, 'sql_time_limit_ms': 1000, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'hash_urls': False, 'allow_facet': True, 'allow_download': True, 'suggest_facets': True, 'default_cache_ttl': 5, 'default_cache_ttl_hashed': 31536000, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'base_url': '/'}

In [4]: (await ds.client.get("/fixtures/facetable.json?_shape=array")).json() Out[4]: [{'pk': 1, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', 'city_id': 1, 'neighborhood': 'Mission', 'tags': '["tag1", "tag2"]', 'complex_array': '[{"foo": "bar"}]', 'distinct_some_null': 'one'}, {'pk': 2, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', 'city_id': 1, 'neighborhood': 'Dogpatch', 'tags': '["tag1", "tag3"]', 'complex_array': '[]', 'distinct_some_null': 'two'}, ```

Huh... maybe I don't need to do anything here? It looks like it's been kept up to date: https://github.com/Homebrew/homebrew-core/commits/master/Formula/sqlite-utils.rb

Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#transforming-tables

.transform() has landed now which should make this a lot easier to solve.

Problem: extract means something else now, see #47 and the upcoming work in #42.

Confirmed this is a bug, https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes explicitly says you should do the PRAGMA foreign_keys bits before and after the transaction, not during.

Right now my code does this INSIDE the transaction: https://github.com/simonw/sqlite-utils/blob/f29f6821f2d08e91c5c6d65d885a1bbc0c743bdd/sqlite_utils/db.py#L790-L793

Idea for CLI options: --type age integer --drop colname --rename oldname newname --not-null col --not-null-false col --pk new_id --pk-none --default col value --default-none column --drop-foreign-key col other_table other_column

Documentation for the .transform() method #114 (now landed) is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table

Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table

One last thing. https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_change says that the first step should be:

If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF.

And the last steps should be:

If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints.

Commit the transaction started in step 2.

If foreign keys constraints were originally enabled, reenable them now.

I need to implement that.

It would be neat if .transform(pk=None) converted a primary key table to a rowid table.

Just needs documentation now.

Design decision needed on foreign keys: what does the syntax look like for removing an existing foreign key?

Since I already have a good implementation of add_foreign_key() I'm tempted to only support dropping them. Maybe like this:

python table.transform(drop_foreign_keys=[("author_id", "author", "id")]) It's a bit crufty but it's such a rare use-case that I think this will be good enough.

I'm not going to do conversions= because it would be inconsistent with how they work elsewhere. The SQL generated by this function looks like this:

INSERT INTO dogs_new_tmp VALUES (a, b) SELECT a, b from dogs;

So passing conversions={"name": "upper(?)"}) wouldn't make sense, since we're not using arguments hence there is no-where for that ? to go.

Maybe something like this:

sqlite-utils transform mydb.db mytable -c age integer --rename age dog_age

This would apply to .transform() in #114 too.

The reason I'm working on this now is that I'd like to support many more options for data cleanup in the Datasette ecosystem - so being able to do things like convert the type of existing columns becomes increasingly important.

Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#registering-custom-sql-functions

Inspired by the idea of adding conversions= to #114 - since this would make it easy to register custom Python functions that can be used to convert the values in a table.

Idea: allow a conversions= parameter, as seen on .insert_all() and friends, which lets you apply a SQL transformation function as part of the operation. E.g.:

python table.transform({"age": int}, conversions={"name": "upper(?)"})

https://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions

A less horrible interface might be the following: ```python

Ensure the 'age' column is not null:

table.transform(not_null={"age"})

The 'age' column is not null but I don't want it to be:

table.transform(not_null={"age": False}) ``` So if the argument is a set it means "make sure these are all not null" - if the argument is a dictionary it means "set these to be null or not null depending on if their dictionary value is true or false".

Syntax challenge: I could use .transform(defaults={"age": None}) to indicate that the age column should have its default removed, but how would I tell .transform() that the age column, currently not null, should have the not null removed from it?

I could do this: .transform(not_not_null={"age"}) - it's a bit gross but it's also kind of funny. I actually like it!

I should support not_null= and default= arguments to the .transform() method because it looks like you can't use ALTER TABLE to change those.

Good news: the .columns introspection does tell me those things: ```

import sqlite_utils db = sqlite_utils.Database(memory=True) db.create_table("foo", {"id": int, "name": str, "age": int}, defaults={"age": 1}, not_null={"name", "age"})

How would I detect which columns are not_null and what their defaults are? I don`t think my introspection logic handles that yet.

When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about hash_id or extracts= since those don't affect the structure of the table as it is being created - well, hash_id does but if we are transforming an existing table we will get the hash_id column for free.

I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787