issue_comments

I'm going to have to untangle Datasette's error handling a bit for this - currently the expectation is that exceptions will be handled at a higher level, but I need to rethink that to make it cleaner for views like the "execute custom SQL" view to add their own error handling (and still be able to return the correct HTTP status codes, even with custom pages).

I'll add this to the succesful JSON format: json { "ok": true, "error": null }

So the JSON (still served with a 500 code) will look something like this: json { "ok": false, "status": 500, "database": "fixtures", "query_name": null, "rows": [], "truncated": false, "error": "Error message goes here", "columns": [], "query": { "sql": "the query that broke goes here", "params": {} }, "private": false, "allow_execute_sql": true, "query_ms": 0.8716583251953125, "source": "tests/fixtures.py", "source_url": "https://github.com/simonw/datasette/blob/master/tests/fixtures.py", "license": "Apache License 2.0", "license_url": "https://github.com/simonw/datasette/blob/master/LICENSE" }

This is a little tricky to solve, because of the location of the form and the need to return JSON as well as HTML. It would be weird if a JSON request came in and got back the standard output from https://latest.datasette.io/fixtures.json when they were expecting to get back JSON in the shape of https://latest.datasette.io/fixtures.json?sql=select%20*%20from%20sqlite_master

I'm going to return the HTML view that you would get for 0 results for a query - https://latest.datasette.io/fixtures?sql=select%201%20limit%200 - but with an error message.

Yeah that sucks. Bumping this up the priority list.

There's something to be said for making this operation pausable and resumable, especially if I'm going to make it available in a Datasette plugin at some point.

Also what would happen if the table had new rows added to it while that command was running?

Problem with this approach is it's not compatible with progress bars - but if it's a multiple of times faster it's worth it.

I wonder if I could make this faster by separating it out into a few steps: - Create the new lookup table with all of the distinct rows - Add the blank foreign key column - run a UPDATE table SET blah_id = (select id from lookup where thang = table.thang) - Drop the value columns

I ran sudo py-spy top -p 123 against the process while it was running and the most time is definitely spent in .update(): ``` Total Samples 1000 GIL: 0.00%, Active: 90.00%, Threads: 1

%Own %Total OwnTime TotalTime Function (filename:line)
38.00% 38.00% 3.85s 3.85s update (sqlite_utils/db.py:1283) 27.00% 27.00% 2.12s 2.12s execute (sqlite_utils/db.py:161) 10.00% 10.00% 0.890s 0.890s execute (sqlite_utils/db.py:163) 10.00% 17.00% 0.870s 1.54s columns (sqlite_utils/db.py:553) 0.00% 0.00% 0.110s 0.210s <listcomp> (sqlite_utils/db.py:554) 0.00% 3.00% 0.100s 0.320s table_names (sqlite_utils/db.py:191) 0.00% 0.00% 0.100s 0.100s new (<string>:1) ```

This can only work when it's reading from a file, not when it's reading from standard input.

This is still a good idea.

lookup_table.lookup(lookups) is doing a SQL lookup. This could be cached in-memory, maybe with a LRU cache, to avoid looking up the primary key for records that we have recently used.

The .update() method it is calling first does a get() and then does a SQL UPDATE ... WHERE:

https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L1244-L1264

Batching those updates may have an effect. Or finding a way to skip the .get() since we already know we have a valid record.

Here's the loop that's taking the time: https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L892-L897

Steps to produce that database: curl -o salaries.csv 'https://data.sfgov.org/api/views/88g8-5mnd/rows.csv?accessType=DOWNLOAD' sqlite-utils insert salaries.db salaries salaries.csv --csv

@simonw @db.register_function decorator, closes #162 4824775 @simonw table.transform() method - closes #114 987dd12 @simonw Keyword only arguments for transform() f8e10df

Also renamed columns= to types=

Closes #165

Commits on Sep 22, 2020 @simonw Implemented sqlite-utils transform command, closes #164 752d261 @simonw Applied Black f29f682 @simonw table.extract() method, refs #42 f855379 @simonw Docstring for sqlite-utils transform c755f28 @simonw Added table.extract(rename=) option, refs #42 c3210f2 @simonw Applied Black 317071a @simonw New .rows_where(select=) argument 7178231 @simonw table.extract() now works with rowid tables, refs #42 2db6c5b @simonw sqlite-utils extract, closes #42 55cf928 @simonw Progress bar for "sqlite-utils extract", closes #169 5c4d58d @simonw Fixed PRAGMA foreign_keys handling for .transform, closes #167