issues

Table actions

2 rows where state = "closed", type = "issue" and user = 15178711 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

repo 2

type 1

  • issue · 2 ✖

state 1

  • closed · 2 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1801394744 I_kwDOCGYnMM5rXxo4 567 Plugin system asg017 15178711 closed 0     9 2023-07-12T17:02:14Z 2023-07-22T22:59:37Z 2023-07-22T22:59:36Z CONTRIBUTOR  

I'd like there to be a plugin system for sqlite-utils, similar to the datasette/llm plugins. I'd like to make plugins that would do things like:

  • Register SQLite extensions for more SQL functions + virtual tables
  • Register new subcommands
  • Different input file formats for sqlite-utils memory
  • Different output file formats (in addition to --csv --tsv --nl etc.

A few real-world use-cases of plugins I'd like to see in sqlite-utils:

  • Register many of my sqlite extensions in sqlite-utils (sqlite-http, sqlite-lines, sqlite-regex, etc.)
  • New subcommands to work with sqlite-vss vector tables
  • Input/ouput Parquet/Avro/Arrow IPC files with sqlite-arrow
 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/567/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1339663518 I_kwDOBm6k_c5P2aSe 1784 Include "entrypoint" option on `--load-extension`? asg017 15178711 closed 0     2 2022-08-16T00:22:57Z 2022-08-23T18:34:31Z 2022-08-23T18:34:31Z CONTRIBUTOR  

Problem

SQLite extensions have the option to define multiple "entrypoints" in each loadable extension. For example, the upcoming version of sqlite-lines will have 2 entrypoints: the default sqlite3_lines_init (which SQLite will automatically guess for) and sqlite3_lines_noread_init. The sqlite3_lines_noread_init version omits functions that read from the filesystem, which is necessary for security purposes when running untrusted SQL (which Datasette does).

(Similar multiple entrypoints will also be added for sqlite-http).

The --load-extension flag, however, doesn't give the option to specify a different entrypoint, so the default one is always used.

Proposal

I want there to be a new command line option of the --load-extension flag to specify a custom entrypoint like so: datasette my.db \ --load-extension ./lines0 sqlite3_lines0_noread_init

Then, under the hood, this line of code:

https://github.com/simonw/datasette/blob/7af67b54b7d9bca43e948510fc62f6db2b748fa8/datasette/app.py#L562

Would look something like this:

python conn.execute("SELECT load_extension(?, ?)", [extension, entrypoint])

One potential problem: For backward compatibility, I'm not sure if Click allows cli flags to have variable number of options ("arity"). So I guess it could also use a : delimiter like --static:

datasette my.db \ --load-extension ./lines0:sqlite3_lines0_noread_init

Or maybe even a new flag name?

datasette my.db \ --load-extension-entrypoint ./lines0 sqlite3_lines0_noread_init

Personally I prefer the : option... and maybe even --load-extension -> --load? Definitely out of scope for this issue tho

datasette my.db \ --load./lines0:sqlite3_lines0_noread_init

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1784/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);