issues

Closes #1983

All documentation here. Edits welcome.

:books: Documentation preview :books:: https://datasette--1996.org.readthedocs.build/en/1996/

Two methods will get canned queries for a Datasette instance:

Datasette.get_canned_queries will return all canned queries for a database that an actor can see.

Datasette.get_canned_query will return a single canned query by name.

Code is here: https://github.com/simonw/datasette/blob/main/datasette/app.py#L503

This will be the official way to access metadata from plugins.

Posting this while I'm thinking about it: I mentioned at the end of this thread that I'm usually doing datasette --get to export canned queries.

I used to use a tool called datafreeze to do scripted exports, but that project looks dead now. The ergonomics of it are pretty nice, though, and the Freezefile.yml structure is actually not too far from Datasette's canned queries.

This is related to the idea for datasette query (#1356) but I think it's a distinct feature. It's most likely a plugin, but I want to raise it here because it's probably something other people have thought about.

Ran inspect on a spatialite database and got these warnings:

ERROR: conn=<sqlite3.Connection object at 0x119e46110>, sql = 'select count(*) from [SpatialIndex]', params = None: no such module: VirtualSpatialIndex ERROR: conn=<sqlite3.Connection object at 0x119e46110>, sql = 'select count(*) from [ElementaryGeometries]', params = None: no such module: VirtualElementary ERROR: conn=<sqlite3.Connection object at 0x119e46110>, sql = 'select count(*) from [KNN]', params = None: no such module: VirtualKNN

It still worked, but probably want to catch this.

I started sketching out a plugin to add a datasette query subcommand to export data from the command line. This is based on discussions in #1356 and #1605. Before I get too far down this rabbit hole, I figure it's worth getting some feedback here (unless this should happen in Discussions). Here's what I'm thinking:

At its most basic, it will write the results of a query to STDOUT.

sh datasette query -d data.db 'select * from data' > results.json

This isn't much improvement over using sqlite-utils. To make better use of datasette and its ecosystem, run datasette query using a canned query defined in a metadata.yml file.

For example, using the metadata file from alltheplaces-datasette:

sh cd alltheplaces-datasette datasette query -d alltheplaces.db -m metadata.yml count_by_spider

That query would be good to get as CSV, and we can auto-discover metadata and databases in the current directory:

sh cd alltheplaces-datasette datasette query count_by_spider -f csv

In this case, count_by_spider is a canned query defined on the alltheplaces database. If the same query is defined on multiple databases or its otherwise unclear which database query should use, pass the -d or --database option.

If a query takes parameters, I can pass them in at runtime, using the --param or -p option:

sh datasette query -d data.db -p value something 'select * from neighborhoods where some_column = :value'

I'm very interested in feedback on this, including whether it should be a plugin or in Datasette core. (I don't have a strong opinion about this, but I'm prototyping it as a plugin to start.)

This is a fairly new feature -- SQLite version 3.31.0 (2020-01-22) -- that I, admittedly, haven't gotten to work yet. But it looks incredibly useful: https://dgl.cx/2020/06/sqlite-json-support

I'm not sure if this is an option on add-column or a separate command like add-generated-column. Either way, it needs an argument to populate it. It could be something like this:

sh sqlite-utils add-column data.db table-name generated --as 'json_extract(data, "$.field")' --virtual

More here: https://www.sqlite.org/gencol.html

Following our conversation last week, want to note this here before I forget.

I've had a couple situations where I'd like to do a bunch of updates in an async event loop, but I run into SQLite's issues with concurrent writes. This feels like something sqlite-utils could help with.

PeeWee ORM has a SQLite write queue that might be a good model. It's using threads or gevent, but I think that approach would translate well enough to asyncio.

Happy to help with this, too.