• 458 - the register_function(name=...) argument

• New tutorial: Cleaning data with sqlite-utils and Datasette provides a tutorial introduction (and accompanying ten minute video) about using this tool.
• New Discord community, https://discord.gg/Ass7bCAMDw

• 469 sqlite-utils rows --order option

• 471 sqlite-utils query --functions option

• 472 Improved code compilation pattern

• 473 Support entrypoints for --load-extension

• 455

• 475 table.default_values property

• 467

Documentation:

• https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table
• https://sqlite-utils.datasette.io/en/latest/cli.html#creating-tables

Documentation: - https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table - https://sqlite-utils--468.org.readthedocs.build/en/468/cli.html#creating-tables

Codecov Report

Merging #468 (fc38480) into main (c5f8a2e) will decrease coverage by 0.12%. The diff coverage is 88.88%.

:exclamation: Current head fc38480 differs from pull request most recent head 2f6a64f. Consider uploading reports for the commit 2f6a64f to get more accurate results

```diff @@ Coverage Diff @@

main #468 +/-

========================================== - Coverage 96.60% 96.47% -0.13%
========================================== Files 6 6
Lines 2590 2640 +50
========================================== + Hits 2502 2547 +45
- Misses 88 93 +5
```

| Impacted Files | Coverage Δ | | |---|---|---| | sqlite_utils/cli.py | 95.86% <66.66%> (-0.09%) | :arrow_down: | | sqlite_utils/db.py | 97.31% <90.19%> (-0.23%) | :arrow_down: |

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Just needs documentation now, which can go here: https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table

Documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#default-values - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.default_values

Interesting challenge with default_value: I need to be able to tell if the default values passed to .create() differ from those in the database already.

Introspecting that is a bit tricky:

```pycon

import sqlite_utils db = sqlite_utils.Database(memory=True) db["blah"].create({"id": int, "name": str}, not_null=("name",), defaults={"name": "bob"})

For the moment I'm not going to pay attention to foreign_keys changes - I will note that these are not modified in the documentation.

The main challenge here is coming up with comprehensive tests. The cases I need to cover are from this block of code:

https://github.com/simonw/sqlite-utils/blob/c5f8a2eb1a81a18b52825cc649112f71fe419b12/sqlite_utils/db.py#L1468-L1480

A couple of tricks I use here. Firstly, I often create the table before the import using the sqlite-utils create-table command: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-tables

The other current option is to use the bulk command, which lets you construct a custom SQL query to execute against every row from a CSV file: https://sqlite-utils.datasette.io/en/stable/cli.html#executing-sql-in-bulk

Do either of those options work here or is there a useful new feature that would work better?

I could add a --transform option to sqlite-utils create-table too.

I did not know is worked against strings and integers and booleans!

https://latest.datasette.io/_memory?sql=select+1+is+1%2C+2.3+is+2.3%2C+true+is+true%2C+true+is+false%2C+%27dog%27+is+%27dog%27%2C+%27dog%27+is+%27cat%27%2C+null+is+null%2C+null+is+not+null

sql select 1 is 1, 2.3 is 2.3, true is true, true is false, 'dog' is 'dog', 'dog' is 'cat', null is null, null is not null

| 1 is 1 | 2.3 is 2.3 | true is true | true is false | 'dog' is 'dog' | 'dog' is 'cat' | null is null | null is not null | |----------|--------------|----------------|-----------------|------------------|------------------|----------------|--------------------| | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |

Codecov Report

Merging #473 (72db599) into main (a46a5e3) will decrease coverage by 0.07%. The diff coverage is 50.00%.

```diff @@ Coverage Diff @@

main #473 +/-

========================================== - Coverage 96.67% 96.60% -0.08%
========================================== Files 6 6
Lines 2587 2590 +3
========================================== + Hits 2501 2502 +1
- Misses 86 88 +2
```

| Impacted Files | Coverage Δ | | |---|---|---| | sqlite_utils/cli.py | 95.94% <50.00%> (-0.18%) | :arrow_down: |

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

cc @asg017 since this feature was inspired by his work on Datasette.

Tests passed in CI against Ubuntu: tests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] PASSED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] PASSED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] PASSED [ 19%]

To run the tests locally on my Mac I needed to run Ubuntu in Docker (for the compilation step).

This worked:

docker run -it -v "$(pwd):/app" ubuntu:latest /bin/bash

Then:

apt-get update && apt-get install python3-pip gcc libsqlite3-dev -y cd /app/tests gcc ext.c -fPIC -shared -o ext.so cd /app pip3 install -e '.[test]' pytest -k test_load_extension

Extension looks to be correctly compiled:

Run (cd tests && gcc ext.c -fPIC -shared -o ext.so && ls -lah) total 412K drwxr-xr-x 3 runner docker 4.0K Aug 27 05:40 . drwxr-xr-x 8 runner docker 4.0K Aug 27 05:40 .. -rw-r--r-- 1 runner docker 0 Aug 27 05:40 __init__.py -rw-r--r-- 1 runner docker 760 Aug 27 05:40 conftest.py -rw-r--r-- 1 runner docker 1.6K Aug 27 05:40 ext.c -rwxr-xr-x 1 runner docker 16K Aug 27 05:40 ext.so drwxr-xr-x 2 runner docker 4.0K Aug 27 05:40 sniff So why doesn't this check pass?

https://github.com/simonw/sqlite-utils/blob/3f694e51a0212a297f663e2fc7f9cdad8775a550/tests/test_cli.py#L25-L30

That didn't run the tests as expected on Ubuntu: https://github.com/simonw/sqlite-utils/runs/8048229213?check_suite_focus=true

tests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] SKIPPED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] SKIPPED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] SKIPPED [ 19%]

That should have worked because of this: https://github.com/simonw/sqlite-utils/blob/3f694e51a0212a297f663e2fc7f9cdad8775a550/.github/workflows/test.yml#L38-L41

Here's the docs I get to update to remove the documented global hack: https://sqlite-utils.datasette.io/en/3.28/cli.html#cli-convert-complex

Annoyingly this means the --import option is no longer required, but removing it would be a backwards incompatible breakage so I need to leave it in. I can minimize it in the documentation though.

Testing bulk --functions:

% sqlite-utils create-table chickens.db chickens id integer name text name_upper text % echo 'id,name 1,Blue 2,Snowy 3,Azi 4,Lila 5,Suna 6,Cardi' | sqlite-utils bulk chickens.db ' insert into chickens (id, name, name_upper) values (:id, :name, myupper(:name)) ' - --functions ' def myupper(s): return s.upper() ' --csv % sqlite-utils rows chickens.db chickens [{"id": 1, "name": "Blue", "name_upper": "BLUE"}, {"id": 2, "name": "Snowy", "name_upper": "SNOWY"}, {"id": 3, "name": "Azi", "name_upper": "AZI"}, {"id": 4, "name": "Lila", "name_upper": "LILA"}, {"id": 5, "name": "Suna", "name_upper": "SUNA"}, {"id": 6, "name": "Cardi", "name_upper": "CARDI"}]

I don't think I need separate documentation for sqlite-utils memory and sqlite-tils bulk since they work the same, and the --help text provides the necessary hints.

Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#defining-custom-sql-functions

I should add this --functions feature to the memory and bulk commands too.

Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table

Found a fix for that!

I replaced this:

python locals = {} globals = {} exec(functions, globals, locals) # Register all callables in the locals dict: for name, value in locals.items(): if callable(value): db.register_function(value, name=name) With this: python globals = {} exec(functions, globals) # Register all callables in the globals dict: for name, value in globals.items(): if callable(value): db.register_function(value, name=name) Because https://docs.python.org/3/library/functions.html#exec says:

If only globals is provided, it must be a dictionary (and not a subclass of dictionary), which will be used for both the global and the local variables. If globals and locals are given, they are used for the global and local variables, respectively.

This works: ``` sqlite-utils :memory: 'select extract_domain("https://www.google.com/blah")' --functions ' from urllib.parse import urlparse

def extract_domain(url): global urlparse return urlparse(url).netloc ' ```

I've encountered this problem before: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-convert-complex

``` $ sqlite-utils convert content.db articles score ' import random random.seed(10)

def convert(value): global random return random.random() ' `` Note theglobal randomline here. Due to the way the tool compiles Python code, this is necessary to ensure therandommodule is available within theconvert()function. If you were to omit this you would see aNameError: name 'random' is not defined` error.

I tried this: ``` sqlite-utils :memory: 'select extract_domain("https://www.google.com/blah")' --functions ' from urllib.parse import urlparse

def extract_domain(url): from urllib.parse import urlparse return urlparse(url).netloc ' And got: NameError: name 'urlparse' is not defined Error: user-defined function raised exception But this worked OK: % sqlite-utils :memory: 'select extract_domain("https://www.google.com/blah")' --functions ' def extract_domain(url): from urllib.parse import urlparse return urlparse(url).netloc ' [{"extract_domain(\"https://www.google.com/blah\")": "www.google.com"}] ```

Quick prototype: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 43e76fa..5dee4f6 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1633,6 +1633,9 @@ def drop_view(path, view, ignore, load_extension): type=(str, str), help="Named :parameters for SQL query", ) +@click.option( + "--functions", help="Python code defining one or more custom SQL functions" +) @load_extension_option def query( path, @@ -1649,6 +1652,7 @@ def query( raw, param, load_extension, + functions, ): """Execute SQL query and return the results as JSON

@@ -1665,6 +1669,16 @@ def query( _load_extensions(db, load_extension) db.register_fts4_bm25()

• Register any Python functions as SQL functions:

• if functions:
• locals = {}
• globals = {}
• exec(functions, globals, locals)

• Register all callables in the locals dict:

• for name, value in locals.items():
• if callable(value):
• db.register_function(value, name=name) + _execute_query( db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols ) Demo:bash % sqlite-utils :memory: 'select 1 + dog()' --functions ' quote> def dog(): quote> return 2 quote> ' [{"1 + dog()": 3}] ```

Alternative design would be --function name definition - like this:

sqlite-utils data.db 'update images set domain = extract_domain(url)' --function extract_domain ' from urllib.parse import urlparse return urlparse(url).netloc ' I like the --functions design better because it leaves space for import statements at the top of the code block, and allows more than one function to be defined in a single go.

I can borrow Alex's example extension for the tests too: https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/tests/ext.c