issue_comments

Just tried this out by adding the example from the docs to my sqlite-utils-hello-world plugin and running: bash sqlite-utils install -e ../sqlite-utils-hello-world Obtaining file:///Users/simon/Dropbox/Development/sqlite-utils-hello-world ... Then: bash sqlite-utils memory "select hello('simon')" json [{"hello('simon')": "Hello, simon!"}] Also: ```pycon

import sqlite_utils db = sqlite_utils.Database(memory=True) list(db.query("select hello('simon')")) [{"hello('simon')": 'Hello, simon!'}] db2 = sqlite_utils.Database(memory=True, execute_plugins=False) list(db2.query("select hello('simon')")) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 494, in query cursor = self.execute(sql, params or tuple()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 512, in execute return self.conn.execute(sql, parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sqlean.dbapi2.OperationalError: no such function: hello ```

Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#prepare-connection-conn

Documented here: https://sqlite-utils.datasette.io/en/latest/plugins.html

Relevant code: https://github.com/simonw/sqlite-utils/blob/3f80a026983d3e634f05a46f2a6da162b5139dd9/sqlite_utils/db.py#L346

Now that we have two plugin hooks I'm closing this issue (we can open other issues for further hooks).

https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.03 :tada:

Comparison is base (86a352f) 96.09% compared to head (faf398f) 96.12%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Glitch in the rendered documentation from https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn

Alex built this in: - #573

Splitting off an issue for prepare_connection() since Alex got the PR in seconds before I shipped 3.34!

I was literally seconds away from shipping version 3.34 but I this looks good so I'm going to try and get it in there.

Demo: ```pycon

from sqlite_utils import Database db = Database(memory=True) db["foo"].insert({"name": "Cleo"})

I'm happy with how this works on https://sqlite-utils.datasette.io/en/stable/cli-reference.html

Found it: https://github.com/simonw/sqlite-utils/blob/18f190e28334d821be78a1dbbf31d7610fc1f9c1/.github/workflows/test.yml#L29-L31

I don't know where that is coming from! I can't see textual as a dependency we are pulling in anywhere.

CLI documentation:

• https://sqlite-utils.datasette.io/en/latest/cli.html#renaming-a-table
• https://sqlite-utils.datasette.io/en/latest/cli-reference.html#rename-table

Python method documentation: - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.rename_table - https://sqlite-utils.datasette.io/en/latest/python-api.html#renaming-a-table

Manually testing new rename-table command: $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT ); $ sqlite-utils rename-table /tmp/f.db bad-table hi Error: Table "bad-table" could not be renamed. no such table: bad-table $ sqlite-utils rename-table /tmp/f.db foo foo Error: Table "foo" could not be renamed. there is already another table or index with this name: foo $ sqlite-utils rename-table /tmp/f.db foo bar $ sqlite-utils schema /tmp/f.db CREATE TABLE "bar" ( [id] INTEGER, [name] TEXT );

Also need a design for an option for the .transform() method to indicate that the new table should be created with a new name without dropping the old one.

I think keep_table="name_of_table" is good for this.

Cool. I might try to add a geojson plugin that handles both input and output. That would help me out a lot.

Trying out a simple first implementation: ```pycon

from sqlite_utils import Database db = Database(memory=True, tracer=print) PRAGMA recursive_triggers=on; None db["foo"].insert({"id": 1}) select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None CREATE TABLE [foo] ( [id] INTEGER ); None select name from sqlite_master where type = 'view' None INSERT INTO [foo] ([id]) VALUES (?); [1] select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'table' None PRAGMA table_info([foo]) None

Here's why that test broke: https://github.com/simonw/sqlite-utils/blob/58b577279fcd5ef6ce88f88b28668dffebfe7f44/sqlite_utils/db.py#L960-L964

I added an extra if self[name].exists() check to the db.create_table() method.

I think this broke a test: ``` $ pytest tests/test_tracer.py
=============================================== test session starts ================================================ platform darwin -- Python 3.11.4, pytest-7.2.2, pluggy-1.0.0 rootdir: /Users/simon/Dropbox/Development/sqlite-utils plugins: icdiff-0.6, hypothesis-6.68.2 collected 2 items

tests/test_tracer.py F. [100%]

===================================================== FAILURES ===================================================== _________ test_tracer __________

def test_tracer():
    collected = []
    db = Database(
        memory=True, tracer=lambda sql, params: collected.append((sql, params))
    )
    db["dogs"].insert({"name": "Cleopaws"})
    db["dogs"].enable_fts(["name"])
    db["dogs"].search("Cleopaws")

  assert collected == [
        ("PRAGMA recursive_triggers=on;", None),
        ("select name from sqlite_master where type = 'view'", None),
        ("select name from sqlite_master where type = 'table'", None),
        ("select name from sqlite_master where type = 'view'", None),
        ("CREATE TABLE [dogs] (\n   [name] TEXT\n);\n        ", None),
        ("select name from sqlite_master where type = 'view'", None),
        ("INSERT INTO [dogs] ([name]) VALUES (?);", ["Cleopaws"]),
        ("select name from sqlite_master where type = 'view'", None),
        (
            "CREATE VIRTUAL TABLE [dogs_fts] USING FTS5 (\n    [name],\n    content=[dogs]\n)",
            None,
        ),
        (
            "INSERT INTO [dogs_fts] (rowid, [name])\n    SELECT rowid, [name] FROM [dogs];",
            None,
        ),
        ("select name from sqlite_master where type = 'view'", None),
    ]

E assert equals failed E [ [
E ('PRAGMA recursive_triggers=on;', None), ('PRAGMA recursive_triggers=on;', None),
E (
E "select name from sqlite_master where type =
E 'view'",
E None, ... E
E ...Full output truncated (13 lines hidden), use '-vv' to show

tests/test_tracer.py:12: AssertionError ============================================= short test summary info ============================================== FAILED tests/test_tracer.py::test_tracer - assert equals failed =========================================== 1 failed, 1 passed in 0.05s ============================================ ```

Here's documentation for the new plugins mechanism, including a very short tutorial on writing a new plugin (inspired by https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html):

https://sqlite-utils.datasette.io/en/latest/plugins.html

Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#register-commands-cli

Manual testing of CLI command as well: $ sqlite-utils create-table /tmp/f.db foo id integer $ sqlite-utils create-table /tmp/f.db foo id integer Error: Table "foo" already exists. Use --replace to delete and replace it. $ sqlite-utils create-table /tmp/f.db foo id integer --replace $ sqlite-utils create-table /tmp/f.db foo id $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER ); $ sqlite-utils create-table /tmp/f.db foo id integer name str --transform Error: column types must be one of ('INTEGER', 'TEXT', 'FLOAT', 'BLOB') $ sqlite-utils create-table /tmp/f.db foo id integer name text --transform $ sqlite-utils schema /tmp/f.db CREATE TABLE "foo" ( [id] INTEGER, [name] TEXT ); $ sqlite-utils create-table /tmp/f.db foo id integer name text --ignore $ sqlite-utils create-table /tmp/f.db foo id integer name text --replace $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT );

Demo: ```pycon

from sqlite_utils import Database db = Database(memory=True) db["foo"].create({"id": int})

I think this is replace=True and ignore=True to match the CLI. And refactoring the CLI to use them.

Tested this locally with: bash datasette install -e ../datasette-graphql Running datasette plugins confirmed that the plugin had been installed.

Here's the diff for adding it to sqlite-utils: https://github.com/simonw/sqlite-utils/commit/ef31210bf06f920e0890e171c3198f0b0dc8d72d

Example plugin: https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf

bash sqlite-utils install https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf/archive/cd7960e476c441a3c5f619e2a44a641a39b91467.zip pyproject.toml: ```toml [project] name = "sqlite-utils-hello-world" version = "0.1"

[project.entry-points.sqlite_utils] hello_world = "sqlite_utils_hello_world" `sqlite_utils_hello_world.py`:python import click import sqlite_utils

@sqlite_utils.hookimpl def register_commands(cli): @cli.command() def hello_world(): "Say hello world" click.echo("Hello world!") ```

Here's where I added that to LLM: https://github.com/simonw/llm/commit/a396950f7934e82a9968703bb3ce9ab7ab62f7f8

• https://github.com/simonw/llm/issues/49

I'm going to start by adding the register_command hook using the exact same pattern as Datasette and LLM.

Would this possibly make a bunch of x-to-sqlite tools obsolete? Or nudge some to become plugins?

Yeah, it could do! That's not a terrible idea to be honest, those things have really been proliferating.

Alternatively, they could each register themselves as plugins in addition - so if you install e.g. pocket-to-sqlite you could then optionally also run it as sqlite-utils pocket-to-sqlite ...

The benefit there is for people who install sqlite-utils from Homebrew, where it gets its own virtual environment. They could run:

```bash brew install sqlite-utils sqlite-utils install pocket-to-sqlite

sqlite-utils pocket-to-sqlite ... ```

Here's where those are implemented for the create-table CLI command: https://github.com/simonw/sqlite-utils/blob/f7af23837deab5c98dae9441d1f68318065d7d8c/sqlite_utils/cli.py#L1543-L1564

https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table

bash sqlite-utils create-table ... --replace That also has --ignore:

--ignore If table already exists, do nothing --replace If table already exists, replace it --transform If table already exists, try to transform the schema