Column action menus are the cog icons on https://latest.datasette.io/fixtures/facetable

Now live at https://latest.datasette.io/fixtures/facetable

https://user-images.githubusercontent.com/9599/94751748-1f205380-033e-11eb-940b-bb4ee7728d31.png">

Since this menu doesn't provide new functionality, I'm going to ignore the fact that it doesn't exist on portrait mobile view for the moment. Likewise, I'm going to skip making it accessible for the moment since lacking accessibility doesn't prevent functionality from being accessed - the menu-less experience currently works the same as the portrait mobile experience.

I'm going to go based just on the visible values on the current page. I think that's good enough, and it avoids the complexity involved in doing a server-side check for blank values.

I could provide the "Show non-blank values" option only for columns where there are blank values visible on the current page.

This could be a bit confusing though, since the absence of that option could suggest that there are no blank values at all when that's actually not true.

One option: run a separate fetch() call that figures out if any of the columns contain blank values, which gets a bit of extra time to execute. Only show the "Show non-blank values" option in the menu once that has returned.

Bug: https://latest.datasette.io/fixtures/sortable?_sort=sortable

It just ran and didn't find anything. I'll leave it enabled for a while but I may turn it off again, it could be that this kind of Python library isn't really what it's useful for.

A bunch of things to fix:

• It clobbers existing querystring parameters - it needs to leave these alone (but replace the current sort order)
• Facet option should not show up if you are already faceting by that column
• There's no way to close the menu once it has opened!
• Accessibility: SVG icon doesn't even have an alt attribute yet. Should use ARIA when the thing appears.

It's also not visible on mobile, need to think about how that will work.

The arrow icon didn't make sense because I already have a triangle icon showing sort order. I'm trying a cog icon instead:

https://user-images.githubusercontent.com/9599/94743176-ce9ffa80-032b-11eb-8f4d-da3962984a4f.png">

Showing "facet by this" on the primary key column doesn't make sense.

I think I've got everything I need to implement this now.

I had a look around and there isn't an obvious pluggy equivalent in JavaScript world at the moment. Lots of frameworks like jQuery and Vue have their own custom plugin mechanisms.

https://github.com/rekit/js-plugin is a simple standalone plugin mechanism. Not quite as full-featured as Pluggy though - in particular I like how Pluggy supports multiple plugins returning results for the same hook that get concatenated into a list of results.

https://css-tricks.com/designing-a-javascript-plugin-system/ has some ideas.

Thanks for the fix!

It looks like http://maps.natalian.org/data.txt is encoded as latin-1, but sqlite-utils assumes utf-8 and hence breaks.

It would be worth improving the error message here. I could also add a --encoding latin-1 option to sqlite-utils insert to help in consuming files that are stored in charsets other than utf-8.

It would be neat to provide a JavaScript plugin hook that plugins can use to add their own options to this menu. No idea what that would look like though.

Maybe use this as the icon:

<svg width="100px" height="100px" viewBox="0 0 255 255">
  <polygon points="0,64 128,191 255,64"/>
</svg>

https://user-images.githubusercontent.com/9599/94734258-d22c8500-031d-11eb-982b-82cde4b1920f.png">

Another potential action:

• Show rows where this is not blank (equivalent to is not blank filter)

This could be displayed conditionally based on if the column is detected to have any blank rows in it?

This is a little related to the error display issue #619 in that both will require some reworking of how the code is structured.

The form needs to switch from GET to POST if the query is a write query. JavaScript can handle this based on the checkbox - if a user does not have JavaScript submitting the form will cause the form action to be changed to POST and the form to be redisplayed.

Prototype:

<style>
body {
font-family: helvetica;
}
.dropdown-menu {
  display: inline-flex;
  border: 1px solid #ccc;
  border-radius: 4px;
  line-height: 1.4;
  font-size: 12px;
  box-shadow: 2px 2px 2px #aaa;
}
.dropdown-menu ul,
.dropdown-menu li {
  list-style-type: none;
  margin: 0;
  padding: 0;
}
.dropdown-menu li {
  border-bottom: 1px solid #ccc;
}
.dropdown-menu li:last-child {
  border: none;
}
.dropdown-menu a:link,
.dropdown-menu a:visited,
.dropdown-menu a:hover,
.dropdown-menu a:focus
.dropdown-menu a:active {
  text-decoration: none;
  display: block;
  padding: 4px 8px 2px 8px;
  color: #222;
  background-color: #fff;
}
.dropdown-menu a:hover {
  background-color: #eee;
}
.hook {
  display: block;
  position: absolute;
  top: 3px;
  left: 12px;
  width: 0; 
  height: 0; 
  border-left: 5px solid transparent;
  border-right: 5px solid transparent;
  border-bottom: 5px solid #ccc;
}
</style>
<div class="dropdown-menu">
<div class="hook"></div>
<ul>
  <li><a href="#">Sort descending</a></li>
  <li><a href="#">Sort ascending</a></li>
  <li><a href="#">Facet by this</a></li>
  <li><a href="#">Count by this</a></li>
</ul>
</div>

https://user-images.githubusercontent.com/9599/94643784-7241ca00-029c-11eb-8554-863fcc255352.png">

I think the accessible way to do this is with absolute positioning - have a menu icon in the <th> which, when clicked, causes the dropdown menu to appear as an absolutely positioned <div> that is not located within the DOM hierarchy of the<th> itself but is positioned to show up in the correct place.

Future version could have expanding out nested side menus that let you do things like "calculate sum/avg for this column against this-other-column".

More options:

<ul>
  <li><a href="#">Sort descending</a></li>
  <li><a href="#">Sort ascending</a></li>
  <li><a href="#">Facet by this</a></li>
  <li><a href="#">Unique values</a></li>
  <li><a href="#">Count unique values</a></li>
</ul>

That fixed it: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_rrule(%27FREQ%3DHOURLY%3BCOUNT%3D5%27)%2C%0D%0A++dateutil_rrule_date(%0D%0A++++%27FREQ%3DDAILY%3BCOUNT%3D3%27%2C%0D%0A++++%271st+jan+2020%27%0D%0A++)%3B

https://user-images.githubusercontent.com/9599/94605680-4d266a80-024e-11eb-8818-bc8bd7958df4.png">

<style>
@media only screen and (max-width: 576px) {

    .rows-and-columns td:nth-of-type(1):before { content: "dateutil_rrule(\000027FREQ=HOURLY;COUNT=5\000027)"; }

    .rows-and-columns td:nth-of-type(2):before { content: "dateutil_rrule_date(\00000A    \000027FREQ=DAILY;COUNT=3\000027,\00000A    \0000271st jan 2020\000027\00000A  )"; }

}
</style>

This time it's because there are newlines in the column header:

<style>
@media only screen and (max-width: 576px) {

    .rows-and-columns td:nth-of-type(1):before { content: "dateutil_rrule(\000027FREQ=HOURLY;COUNT=5\000027)"; }

    .rows-and-columns td:nth-of-type(2):before { content: "dateutil_rrule_date(
\00000A    \000027FREQ=DAILY;COUNT=3\000027,
\00000A    \0000271st jan 2020\000027
\00000A  )"; }

}
</style>

Those need to be escaped somehow.

This would benefit https://github.com/simonw/datasette-import-table - which currently ignores the CREATE TABLE and derives the schema by inserting rows.

Here's the code that adds it to the HTML context: https://github.com/simonw/datasette/blob/c11383e6284e000b2641569457efa16ac9e0d6ae/datasette/views/table.py#L835-L837

def escape_css_string(s):
    return _css_re.sub(lambda m: "\\" + ("{:X}".format(ord(m.group())).zfill(6)), s)

That fixes it:
https://user-images.githubusercontent.com/9599/94493173-c23b6680-01a0-11eb-9468-e972c51b015c.png">

Weirdly even those leading 0s doesn't fix it:

https://user-images.githubusercontent.com/9599/94492937-44775b00-01a0-11eb-9c5f-e991af620404.png">

But... padding to six characters does! See https://www.w3.org/International/questions/qa-escapes

https://user-images.githubusercontent.com/9599/94492988-61139300-01a0-11eb-8304-bffe448c7d2b.png">

In [32]: print('\\' + "{:X}".format(ord('"')).zfill(6))
\000022

print('\\' + "{:X}".format(ord('"')).zfill(4))
\0022

The fix may be to use \0022 instead of \22.

Here's the HTML for the broken example above:

<style>
@media only screen and (max-width: 576px) {

    .rows-and-columns td:nth-of-type(1):before { content: "dateutil_parse(\2210 october 2020 3pm\22)"; }

    .rows-and-columns td:nth-of-type(2):before { content: "dateutil_easter(\222020\22)"; }

    .rows-and-columns td:nth-of-type(3):before { content: "dateutil_parse_fuzzy(\22This is due 10 september\22)"; }

    .rows-and-columns td:nth-of-type(4):before { content: "dateutil_parse(\221/2/2020\22)"; }

    .rows-and-columns td:nth-of-type(5):before { content: "dateutil_parse(\222020-03-04\22)"; }

    .rows-and-columns td:nth-of-type(6):before { content: "dateutil_parse_dayfirst(\222020-03-04\22)"; }

    .rows-and-columns td:nth-of-type(7):before { content: "dateutil_easter(2020)"; }

}
</style>

The glitch affects the ones where the quote is followed by digits.

Looks like a bug in this function: https://github.com/simonw/datasette/blob/1f021c37110fc9019b0ef70062c28c335e568ae2/datasette/utils/__init__.py#L269-L274

Codecov Report

Merging #977 into main will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #977   +/-   ##
=======================================
  Coverage   84.27%   84.27%           
=======================================
  Files          28       28           
  Lines        3847     3847           
=======================================
  Hits         3242     3242           
  Misses        605      605           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a6d0dc...5c01344. Read the comment docs.

Relevant code: https://github.com/simonw/sqlite-utils/blob/94fc62857ee2655a21d85f6dae84b67bbfa5956d/sqlite_utils/db.py#L331-L367

https://hypothesis.readthedocs.io/

Hi Simon,

Thanks so much for all your work on datasette, it's an excellent project and I wish you all the best with it. I particularly enjoyed your talk at the Django London Meetup a short while back.

I've been trying to publish to Heroku from Windows 10 and I was running into this error. I'm not sure why it can't be run without shell=True on Windows but this seems to help. With this change, I am able to publish if I pass in a name to the publish command. When a name is not passed the default of datasette is used and therefore this line here fails (as datasette at heroku already exists) and causes the recession error mentioned above.

https://github.com/simonw/datasette/blob/9a6d0dce282e7fb58c5610e24c74098c923abfdc/datasette/publish/heroku.py#L126

I tried to write a patch for this but I am really struggling with being on Windows (many of the tests seem to fail anyway?), and my lack of knowledge of Mock, so sorry for this. Hope this is of some help.

SQL query for detecting integers:

select
  'contains_non_integer' as result
from
  mytable
where
  cast(cast(mycolumn AS INTEGER) AS TEXT) != mycolumn
limit
  1

This will return a single row with a 1 as soon as it comes across a column that contains a non-integer - so it short circuits quickly on TEXT columns with non-integers in them.

If everything in the column is an integer it will scan the whole thing before returning no rows.

More extensive demo:

select
  value,
  cast(cast(value AS INTEGER) AS TEXT) = value as is_valid_int
from
  (
    select
      '1' as value
    union
    select
      '1.1' as value
    union
    select
      'dog' as value
    union
    select
      null as value
  )

https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+INTEGER%29+AS+TEXT%29+%3D+value+as+is_valid_int%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29

This recipe looks like it might be the way to detect floats:

select
  value,
  cast(cast(value AS REAL) AS TEXT) in (value, value || '.0') as is_valid_float
from
  (
    select
      '1' as value
    union
    select
      '1.1' as value
    union
    select
      'dog' as value
    union
    select
      null as value
  )

Demo: https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+REAL%29+AS+TEXT%29+in+%28value%2C+value+%7C%7C+%27.0%27%29+as+is_valid_float%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29

Posed a question about this on the SQLite forum here: https://sqlite.org/forum/forumpost/ab0dcd66ef

Yeah this works fine, added a new confirmatory test.

I'm using a click.File() at the moment: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L496

I'll need to change that to be something that I can easily measure progress through. Also I should change its name - json_file is a bad name when it sometimes handles csv or tsv instead.

It looks like the argument provided by click.File doesn't provide a way to read the size of the file, so I need to switch that out for a file path instead. https://click.palletsprojects.com/en/7.x/api/#click.Path

Relevant code: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L550-L560

Changing that to track progress through NL-JSON, CSV and TSV shouldn't be too hard.

I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.

Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.

This was implemented in #161.

Documentation: https://sqlite-utils.readthedocs.io/en/stable/cli.html#transforming-tables

Documentation: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#transforming-a-table

This is complex enough new logic that it will need test coverage - specifically covering tables or databases with strange names.

This is a backwards incompatible change, so technically I should bump the major version to 3. I'm not going to do that, because the feature is brand new and the chance that anyone has written code or shell scripts that use it is vanishingly small.

I think I'll call this option --column-order with a shortcut of -o.

Landed that.

Idea: if a database only has a single table, this could open straight to /db/table. If it has multiple tables but a single database it could open straight to /db.

Need to land #174 first.

For reusing the lookup table: I'm going to raise an error if a lookup table exists but without the correct columns. The caller can then add those columns and try again.

I may revert the now unnecessary undocumented tweaks to the .update() method made in 66d506587eba9f0715267d6560b97c1fa44cc781 as well.

The way the lookup table works here differs from the previous implementation. In the previous implementation the usage of .lookup() meant that an existing table would be modified to fit the new purpose. That no longer happens in this version. Need to make a design decision about how this should work.

It should definitely be possible to use an existing lookup table - imagine a database where several tables have a "Departments" column and we want to extract all of those values out to a single shared "Departments" table.

I could also be a bit smarter about transaction handling. I think it may be possible to run this entire operation in a single transaction now.

import functools is no longer needed.

After running through the steps in https://simonwillison.net/2020/Sep/23/sqlite-utils-extract/ I get a table that looks like this:

https://user-images.githubusercontent.com/9599/94116875-666b8900-fe00-11ea-9e97-2b9ccbfeae29.png">

The foreign key columns are all at the end of the table. It would be nicer if they were arranged in the same order as the columns they replaced.

This could do with a little bit more testing - I'm worried there may be column or table name edge cases that are not covered yet. I also need to remove the progress bar code since that no longer makes sense for this implementation.

I wonder if I could make this faster by separating it out into a few steps:

* Create the new lookup table with all of the distinct rows

* Add the blank foreign key column

* run a `UPDATE table SET blah_id = (select id from lookup where thang = table.thang)`

* Drop the value columns

My prototype of this knocked the time down from 10 minutes to 4 seconds, so I think the change is worth it!

% date
sqlite-utils extract salaries.db salaries \
   'Department Code' 'Department' \
  --table 'departments' \
  --fk-column 'department_id' \
  --rename 'Department Code' code \
  --rename 'Department' name
date
sqlite-utils extract salaries.db salaries \
   'Union Code' 'Union' \
  --table 'unions' \
  --fk-column 'union_id' \
  --rename 'Union Code' code \
  --rename 'Union' name
date
sqlite-utils extract salaries.db salaries \
   'Job Family Code' 'Job Family' \
  --table 'job_families' \
  --fk-column 'job_family_id' \
  --rename 'Job Family Code' code \
  --rename 'Job Family' name
date
sqlite-utils extract salaries.db salaries \
   'Job Code' 'Job' \
  --table 'jobs' \
  --fk-column 'job_id' \
  --rename 'Job Code' code \
  --rename 'Job' name
date
Thu Sep 24 00:48:16 PDT 2020

Thu Sep 24 00:48:20 PDT 2020

Thu Sep 24 00:48:24 PDT 2020

Thu Sep 24 00:48:28 PDT 2020

Thu Sep 24 00:48:32 PDT 2020

Oh. Awesome.

On Thu, Sep 24, 2020 at 12:28:53AM -0700, Simon Willison wrote:

@obra there's a plugin for that! https://github.com/simonw/
datasette-upload-csvs

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.*

--

@obra there's a plugin for that! https://github.com/simonw/datasette-upload-csvs

Come to think of it I've noticed that in the logs when it's running on my laptop, definitely worth fixing.

As a half-measure, I'd get value out of being able to upload a CSV and have datasette run csv-to-sqlite on it.

I'm going to have to untangle Datasette's error handling a bit for this - currently the expectation is that exceptions will be handled at a higher level, but I need to rethink that to make it cleaner for views like the "execute custom SQL" view to add their own error handling (and still be able to return the correct HTTP status codes, even with custom pages).

I'll add this to the succesful JSON format:

{
  "ok": true,
  "error": null
}

So the JSON (still served with a 500 code) will look something like this:

{
  "ok": false,
  "status": 500,
  "database": "fixtures",
  "query_name": null,
  "rows": [],
  "truncated": false,
  "error": "Error message goes here",
  "columns": [],
  "query": {
    "sql": "the query that broke goes here",
    "params": {}
  },
  "private": false,
  "allow_execute_sql": true,
  "query_ms": 0.8716583251953125,
  "source": "tests/fixtures.py",
  "source_url": "https://github.com/simonw/datasette/blob/master/tests/fixtures.py",
  "license": "Apache License 2.0",
  "license_url": "https://github.com/simonw/datasette/blob/master/LICENSE"
}

This is a little tricky to solve, because of the location of the form and the need to return JSON as well as HTML. It would be weird if a JSON request came in and got back the standard output from https://latest.datasette.io/fixtures.json when they were expecting to get back JSON in the shape of https://latest.datasette.io/fixtures.json?sql=select%20*%20from%20sqlite_master

I'm going to return the HTML view that you would get for 0 results for a query - https://latest.datasette.io/fixtures?sql=select%201%20limit%200 - but with an error message.

Yeah that sucks. Bumping this up the priority list.

I've just run into this after crafting a complex query and discovered that hitting back loses my query.

Even showing me the whole bad query would be a huge improvement over the current status quo.

There's something to be said for making this operation pausable and resumable, especially if I'm going to make it available in a Datasette plugin at some point.

Also what would happen if the table had new rows added to it while that command was running?

Problem with this approach is it's not compatible with progress bars - but if it's a multiple of times faster it's worth it.

I wonder if I could make this faster by separating it out into a few steps:
- Create the new lookup table with all of the distinct rows
- Add the blank foreign key column
- run a UPDATE table SET blah_id = (select id from lookup where thang = table.thang)
- Drop the value columns

I ran sudo py-spy top -p 123 against the process while it was running and the most time is definitely spent in .update():

Total Samples 1000
GIL: 0.00%, Active: 90.00%, Threads: 1

  %Own   %Total  OwnTime  TotalTime  Function (filename:line)                                                                                                                                  
 38.00%  38.00%    3.85s     3.85s   update (sqlite_utils/db.py:1283)
 27.00%  27.00%    2.12s     2.12s   execute (sqlite_utils/db.py:161)
 10.00%  10.00%   0.890s    0.890s   execute (sqlite_utils/db.py:163)
 10.00%  17.00%   0.870s     1.54s   columns (sqlite_utils/db.py:553)
  0.00%   0.00%   0.110s    0.210s   <listcomp> (sqlite_utils/db.py:554)
  0.00%   3.00%   0.100s    0.320s   table_names (sqlite_utils/db.py:191)
  0.00%   0.00%   0.100s    0.100s   __new__ (<string>:1)

This can only work when it's reading from a file, not when it's reading from standard input.

This is still a good idea.

lookup_table.lookup(lookups) is doing a SQL lookup. This could be cached in-memory, maybe with a LRU cache, to avoid looking up the primary key for records that we have recently used.

The .update() method it is calling first does a get() and then does a SQL UPDATE ... WHERE:

https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L1244-L1264

Batching those updates may have an effect. Or finding a way to skip the .get() since we already know we have a valid record.

Here's the loop that's taking the time: https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L892-L897

Steps to produce that database:

curl -o salaries.csv 'https://data.sfgov.org/api/views/88g8-5mnd/rows.csv?accessType=DOWNLOAD'
sqlite-utils insert salaries.db salaries salaries.csv --csv

Oh wow oh wow. Thanks so much Simon. In an astoundingly rough week, this is a shining jewel. 🤣

@simonw
@db.register_function decorator, closes #162
4824775
@simonw
table.transform() method - closes #114
987dd12
@simonw
Keyword only arguments for transform()
f8e10df

Also renamed columns= to types=

Closes #165

Commits on Sep 22, 2020
@simonw
Implemented sqlite-utils transform command, closes #164
752d261
@simonw
Applied Black
f29f682
@simonw
table.extract() method, refs #42
f855379
@simonw
Docstring for sqlite-utils transform
c755f28
@simonw
Added table.extract(rename=) option, refs #42
c3210f2
@simonw
Applied Black
317071a
@simonw
New .rows_where(select=) argument
7178231
@simonw
table.extract() now works with rowid tables, refs #42
2db6c5b
@simonw
sqlite-utils extract, closes #42
55cf928
@simonw
Progress bar for "sqlite-utils extract", closes #169
5c4d58d
@simonw
Fixed PRAGMA foreign_keys handling for .transform, closes #167

Documentation for sqlite-utils extract: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table

Trying this demo again:

wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv
sqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name

It worked!

The documentation for the .extract() method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table

I tried out the prototype of the CLI on the Global Power Plants data:

wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv
sqlite-utils extract global.db power_plants country country_long

This threw an error because rowid columns are not yet supported. I fixed that like so:

sqlite-utils transform global.db power_plants --rename rowid id
sqlite-utils extract global.db power_plants country country_long

That worked! But it didn't play great with Datasette, because the resulting extracted table had columns country and country_long and neither of those are called name or value or title.

Based on this I need to add rowid table support AND I need to implement the proposed rename= argument for renaming columns on their way into the new table.

The command-line version of this needs to accept a table and one or more columns, then a --table and --fk-column option.

Here's how I'm generating the examples for the documentation:

In [2]: import sqlite_utils

In [3]: db = sqlite_utils.Database(memory=True)

In [4]: db["Trees"].insert({"id": 1, "TreeAddress": "52 Vine St", "CommonName":
   ...: "Palm", "LatinName": "foo"}, pk="id")
Out[4]: <Table Trees (id, TreeAddress, CommonName, LatinName)>

In [5]: db["Trees"].extract(["CommonName", "LatinName"], table="Species", fk_col
   ...: umn="species_id")

In [6]: print(db["Trees"].schema)
CREATE TABLE "Trees" (
   [id] INTEGER PRIMARY KEY,
   [TreeAddress] TEXT,
   [species_id] INTEGER,
   FOREIGN KEY(species_id) REFERENCES Species(id)
)

In [7]: print(db["Species"].schema)
CREATE TABLE [Species] (
   [id] INTEGER PRIMARY KEY,
   [CommonName] TEXT,
   [LatinName] TEXT
)

Need to make sure this works correctly for rowid tables.

What to do if the table already exists? The .lookup() function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.

So .extract() probably takes a batch_size= argument too, which defaults to maybe 1000.

table.extract() can take an optional progress= argument which is a callback which will be used to report progress - called after each batch with (num_done, total). It will get called with (0, total) once at the start to allow progress bars to be initialized. The command-line progress bar will use this.

And if you want to rename some of the columns in the new table:

db["trees"].extract(["common_name", "latin_name"], table="species", rename={"common_name": "name"})

In Python it looks like this:

# Simple case - species column species_id pointing to species table
db["trees"].extract("species")

# Setting a custom table
db["trees"].extract("species", table="Species")

# Custom foreign key column on trees
db["trees"].extract("species", fk_column="species")

# Extracting multiple columns
db["trees"].extract(["common_name", "latin_name"])
# (this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id)

# Or with explicit table (fk_column here defaults to species_id because of the table name)
db["trees"].extract(["common_name", "latin_name"], table="species")

I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the .lookup() mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID.

That integer ID gets written back into the first of the columns that are being transformed. A .transform() call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.

I think the new foreign key column is called company_name_id by default in this example but can be customized by passing --fk-column=xxx

Thinking more about this one:

$ sqlite-utils extract my.db \
    dea_sales company_name company_address \
    --table companies

The goal here is to pull the company name and address pair out into a separate table.

Some questions:
- should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this.
- what should the foreign key column that gets added to the companies table be called?

CLI design idea:

$ sqlite-utils extract my.db \
    dea_sales company_name

Here we just specify the original table and column - the new extracted table will automatically be called "company_name" and will have "id" and "value" columns, by default.

To set a custom extract table:

$ sqlite-utils extract my.db \
    dea_sales company_name \
    --table companies

And for extracting multiple columns and renaming them on the created table, maybe something like this:

$ sqlite-utils extract my.db \
    dea_sales company_name company_address \
    --table companies \
    --column company_name name \
    --column company_address address

Confirmed in local dev:

% datasette fixtures.db --inspect-file inspect.json
Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette", line 11, in <module>
    load_entry_point('datasette', 'console_scripts', 'datasette')()
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 406, in serve
    inspect_data = json.load(open(inspect_file))
TypeError: 'bool' object is not callable