A problem with this is that if you're using --query you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is .csv and plugin formats can't handle full streams, see #1062 and #1177.

I figured out a neat pattern for streaming JSON arrays in this TIL: https://til.simonwillison.net/python/output-json-array-streaming

Right now, I usually have a line in a Makefile like this:

combined.geojson: project.db
    pipenv run datasette project.db --get /project/combined.geojson \
     --load-extension spatialite \
     --setting sql_time_limit_ms 5000 \
     --setting max_returned_rows 20000 \
     -m metadata.yml > $@

That all assumes I've loaded whatever I need into project.db and created a canned query called combined (and then uses datasette-geojson for geojson output).

It works, but as you can see, it's a lot to manage, a lot of boilerplate, and it wasn't obvious how to get there. If there's an error in the canned query, I get an HTML error page, so that's hard to debug. And it's only one query, so each output needs a line like this. Make isn't ideal, either, for that reason.

The thing I really liked with datafreeze was doing templated filenames. I have a project now where I need to export a bunch of litttle geojson files, based on queries, and it would be awesome to be able to do something like this:

databases:
  project:
    queries:
      boundaries:
        sql: "SELECT * FROM boundaries"
        filename: "boundaries/{id}.geojson"
        mode: "item"
        format: geojson

And then do:

datasette freeze -m metadata.yml project.db

For HTML export, maybe there's a template argument, or format: template or something. And that gets you a static site generator, kinda for free.

Oh that's interesting. I was thinking about this from a slightly different angle recently - pondering what a static site generator built on top of Datasette might look like.

Just a sketch at the moment, but I was imagining a YAML configuration file with a SQL query that returns a list of paths - then a tool that runs that query and uses the equivalent of datasette --get to create a static copy of each of those paths.

I think these two ideas can probably be merged. I'd love to know more about how you are solving this right now!

I think both of these are Spatialite specific. They get generated when you first initialize the extension. KNN is actually deprecated in favor of KNN2, as I understand it.

I need to add sqlite_stat1 to the hidden tables too, see:
- https://github.com/simonw/sqlite-utils/issues/366

Good idea - though I'm nervous about accidentally hiding a data_licenses table outside of the SpatiaLite case. I'll only hide that one if SpatiaLite is loaded.

Potential design:

datasette publish cloudrun data.db \
  --service my-service \
  --domain demo.datasette.io

I think I'm OK with calling this --domain even when it is being used with a subdomain.

This will require datasette.io to already have been verified with Google.

Not sure how best to handle the DNS part - maybe print out instructions for the necessary CNAME?

Also people can use a custom base template and link to a custom favicon if they want to.

For the moment then I will hard-code a new favicon and leave it to ASGI plugins if people want to define their own.

I don't think abusing the template loader mechanism for this will work: Jinja provides an API for loading text templates, but I don't think it can be sensibly abused to open binary image files instead. Loaded code is here: https://github.com/pallets/jinja/blob/main/src/jinja2/loaders.py

I'd like this to be customizable. I'm going to load it from the template loading system such that a custom favicon for a specific instance can be dropped in using a file in templates/.

Codecov Report

Merging #1602 (29f5772) into main (cb29119) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1602   +/-   ##
=======================================
  Coverage   92.09%   92.09%           
=======================================
  Files          34       34           
  Lines        4516     4516           
=======================================
  Hits         4159     4159           
  Misses        357      357           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb29119...29f5772. Read the comment docs.

@simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks.

There's actually already a way to move regular Datasette --setting options to a settings.json file thanks to configuration directory mode: https://docs.datasette.io/en/stable/settings.html

Maybe extending that mechanism to handle plugins would be a neat path forward here.

The datasette publish --plugin-secret name setting value option already implements something a bit like this. https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values

It's a bit of a messy hack to compensate for metadata being visible. Maybe I could replace that mechanism with the proposed plugin configuration rethink from this issue.

I still like the debug benefits of making plugin settings public - perhaps add a rule that if a plugin setting has a secret: prefix it gets redacted on a new /-/plugin-settings page.

This is now shipped!

• Upgraded Pluggy dependency to 1.0. #1575
• Now using Plausible analytics for the Datasette documentation.
• The db.execute_write() internals method now defaults to blocking until the write operation has completed. Previously it defaulted to queuing the write and then continuing to run code while the write was in the queue. #1579
• explain query plan is now allowed with varying amounts of white space in the query. #1588
• New CLI reference page showing the output of --help for each of the datasette sub-commands. This lead to several small improvements to the help copy. #1594
• Fixed bug where columns with a underscore prefix could result in unnecessary hidden form fields. #1527

Commits since 0.60a1: https://github.com/simonw/datasette/compare/0.60a1...3664ddd40

OK, I'm going to recommend a workaround for this instead. Here's asgi.py updated to strip the prefix before passing the request on to Datasette:

import pathlib
from asgi_cors import asgi_cors
from channels.routing import URLRouter
from django.urls import re_path
from datasette.app import Datasette


def rewrite_path(app, prefix_to_strip):
    async def rewrite_path_app(scope, receive, send):
        if (
            scope["type"] == "http"
            and "path" in scope
            and scope["path"].startswith(prefix_to_strip)
        ):
            scope["path"] = scope["path"][len(prefix_to_strip) :]
            if "raw_path" in scope:
                scope["raw_path"] = scope["raw_path"][len(prefix_to_strip) :]
        await app(scope, receive, send)

    return rewrite_path_app


datasette_ = Datasette(
    files=["fixtures.db"],
    settings={"base_url": "/datasettes/", "plugins": {}},
)
application = URLRouter(
    [
        re_path(
            r"^datasettes/.*",
            asgi_cors(rewrite_path(datasette_.app(), "/datasettes"), allow_all=True),
        ),
    ]
)

This works on my laptop - please re-open the ticket if it doesn't work for you!

I think this prefixed string mechanism is supposed to prevent the base_url prefix from being applied twice: https://github.com/simonw/datasette/blob/3664ddd400062123e99500d28b160c7944408c1a/datasette/url_builder.py#L9-L16

But with a bit of extra logging all of the inputs to that are NOT prefixed strings:

Urls.path called with: /datasettes/fixtures/compound_three_primary_keys?_sort=content (PrefixedUrlString = False)
  returning /datasettes/datasettes/fixtures/compound_three_primary_keys?_sort=content

So it looks like urls.path(...) is indeed the code responsible for doubling up that /datasettes/ prefix.

Since this is a special case bug for when using Datasette as a library I wonder if a good fix here would be to support something like this:

application = URLRouter([
  re_path(r"^datasettes/.*", asgi_cors(datasette_.app(remove_path_prefix="datasettes/"), allow_all=True)),
])

Bug is fixed on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1

There are four places in the documentation that use .. literalinclude:: at the moment which I can replace - I can actually just link directly to the new https://docs.datasette.io/en/latest/cli-reference.html page instead of embedding the help directly.

Oh! This is because _city_id has a leading underscore (for testing purposes). I think I need to filter out any keys that contain __ in that case.

What happens to columns that contain a __? They shouldn't be reflected in the hidden arguments either - this code is really only supposed to catch things like _where and _m2m_through= and _col - basically most of the list on https://docs.datasette.io/en/stable/json_api.html#special-table-arguments

That's because that page has this unnecessary hidden form field:

<input type="hidden" name="_city_id__gt" value="1">

That field is added by this bit in the template: https://github.com/simonw/datasette/blob/515f8d38ebae203efc15ca79a8b42848276b35e5/datasette/templates/table.html#L119-L122

Which is populated here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/views/table.py#L813-L821

Fixed:

% datasette --help        
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette is an open source multi-tool for exploring and publishing data

  About Datasette: https://datasette.io/
  Full documentation: https://docs.datasette.io/

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect    Generate JSON summary of provided database files
...

% datasette inspect --help
Usage: datasette inspect [OPTIONS] [FILES]...

  Generate JSON summary of provided database files

  This can then be passed to "datasette --inspect-file" to speed up count
  operations against immutable database files.

Options:
  --inspect-file TEXT
  --load-extension TEXT  Path to a SQLite extension to load
  --help                 Show this message and exit.

Copying those in here:

• New plugin hook: filters_from_request(request, database, table, datasette), which runs on the table page and can be used to support new custom query string parameters that modify the SQL query. (#473)
• The number of unique values in a facet is now always displayed. Previously it was only displayed if the user specified ?_facet_size=max. (#1556)
• Fixed bug where ?_facet_array=tags&_facet=tags would only display one of the two selected facets. (#625)
• Facets of type date or array can now be configured in metadata.json, see Facets in metadata.json. Thanks, David Larlet. (#1552)
• New ?_nosuggest=1 parameter for table views, which disables facet suggestion. (#1557)
• Label columns detected for foreign keys are now case-insensitive, so Name or TITLE will be detected in the same way as name or title. (#1544)
• The query string variables exposed by request.args will now include blank strings for arguments such as foo in ?foo=&bar=1 rather than ignoring those parameters entirely. (#1551)
• Database write connections now execute the prepare_connection(conn, database, datasette) plugin hook. (#1564)
• The Datasette() constructor no longer requires the files= argument, and is now documented at Datasette class. (#1563)
• The tracing feature now traces write queries, not just read queries. (#1568)
• Added two methods for writing to the database: await db.execute_write_script(sql, block=False) and await db.execute_write_many(sql, params_seq, block=False). (#1570)
• Made several performance improvements to the database schema introspection code that runs when Datasette first starts up. (#1555)
• Fixed bug where writable canned queries could not be used with custom templates. (#1547)

Seeing as this area of the code has produced so many bugs in the past, I think part of the fix may be to write comprehensive documentation about how routing works for the internals documentation. Doing so might help me figure this bug out!

Most of these are already written for these two alpha releases:
- https://github.com/simonw/datasette/releases/tag/0.60a0
- https://github.com/simonw/datasette/releases/tag/0.60a1

This works now: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable

Challenge: explain the difference between view as in SQL view, and view as in the code that serves TableView / DatabaseView etc.

See also:
- #1412

Going with this for the copy:

Datasette Desktop is a packaged Mac application which bundles Datasette together with Python and allows you to install and run Datasette directly on your laptop. This is the best option for local installation if you are not comfortable using the command line.

https://docs.datasette.io/en/latest/cli-reference.html

I went with a simpler pattern that sqlite-utils because Datasette has a lot less commands: https://github.com/simonw/datasette/blob/4b23f01f3e668c8f2a2f1a294be49f49b4073969/docs/cli-reference.rst#L9-L35

The implementation can be almost exactly the same as this: https://github.com/simonw/sqlite-utils/blame/74586d3cb26fa3cc3412721985ecdc1864c2a31d/docs/cli-reference.rst#L11-L76

I need to do something extra to ensure the output of datasette --help-settings is shown too.

You can already run datasette --help-settings to see detailed help on available settings. Maybe datasette --help-plugin-settings could do the same thing for plugin settings?

Or I could even have available plugin settings show up as a list at the bottom of the datasette --help-settings output - which currently looks like this:

% datasette --help-settings
Settings:
  default_page_size            Default page size for the table view
                               (default=100)
  max_returned_rows            Maximum rows that can be returned from a table
                               or custom query (default=1000)
  num_sql_threads              Number of threads in the thread pool for
                               executing SQLite queries (default=3)
  sql_time_limit_ms            Time limit for a SQL query in milliseconds
                               (default=1000)
  default_facet_size           Number of values to return for requested facets
                               (default=30)
  facet_time_limit_ms          Time limit for calculating a requested facet
                               (default=200)
  facet_suggest_time_limit_ms  Time limit for calculating a suggested facet
                               (default=50)
  hash_urls                    Include DB file contents hash in URLs, for far-
                               future caching (default=False)
  allow_facet                  Allow users to specify columns to facet using
                               ?_facet= parameter (default=True)
  allow_download               Allow users to download the original SQLite
                               database files (default=True)
  suggest_facets               Calculate and display suggested facets
                               (default=True)
  default_cache_ttl            Default HTTP cache TTL (used in Cache-Control:
                               max-age= header) (default=5)
  default_cache_ttl_hashed     Default HTTP cache TTL for hashed URL pages
                               (default=31536000)
  cache_size_kb                SQLite cache size in KB (0 == use SQLite
                               default) (default=0)
  allow_csv_stream             Allow .csv?_stream=1 to download all rows
                               (ignoring max_returned_rows) (default=True)
  max_csv_mb                   Maximum size allowed for CSV export in MB - set
                               0 to disable this limit (default=100)
  truncate_cells_html          Truncate cells longer than this in HTML table
                               view - set 0 to disable (default=2048)
  force_https_urls             Force URLs in API output to always use https://
                               protocol (default=False)
  template_debug               Allow display of template debug information
                               with ?_context=1 (default=False)
  trace_debug                  Allow display of SQL trace debug information
                               with ?_trace=1 (default=False)
  base_url                     Datasette URLs should use this base path
                               (default=/)

Another option: if I make plugin settings a higher level concept in Datasette than they are at the moment, I could allow them to be set either using --options OR using the existing metadata.yml/json mechanism. https://docs.datasette.io/en/stable/plugins.html#plugin-configuration

I want to make changes to that anyway, because I'm increasingly uncomfortable with plugin settings ending up in the "metadata" mechanism.

I try to stick pretty closely to what Click supports, and Click likes you to define options explicitly so that it can display them in the output of --help.

But... that makes me think that actually showing these options in --help is likely a better idea. My --plugin-setting concept would have help that looks something like this:

% datasette serve --help
  ...
  --plugin-setting <TEXT TEXT TEXT>...     Setting for a specified plugin.

That's not great help!

The alternative would be to allow plugins to register their extra options with the command - which would mean the help output could look like this instead:

% datasette serve --help
  ...
  --tiddlywiki-db <TEXT>    Name of database to use for datasette-tiddlywiki

This feels like a good argument to me for plugins to explicitly register their settings. I'm not sure if I should enforce the name-of-plugin- prefix, or if I should allow plugins to define any setting they like. I'm still nervous about plugins over-riding existing or future default options to that command.

One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.

Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (gis.py or something) just so it's easier to find later.

Some proposals I'd add to sqlite-utils:

Some version of this, from geojson-to-sqlite:

def init_spatialite(db, lib):
    db.conn.enable_load_extension(True)
    db.conn.load_extension(lib)
    # Initialize SpatiaLite if not yet initialized
    if "spatial_ref_sys" in db.table_names():
        return
    db.conn.execute("select InitSpatialMetadata(1)")

Also a function for creating a spatial index:

db.conn.execute("select CreateSpatialIndex(?, ?)", [table, "geometry"])

I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like:

sqlite-utils spatial-index spatial.db table-name column-name

Came here to add this. I might pick it up.

Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.

Codecov Report

Merging #1593 (df73ebb) into main (8c401ee) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1593   +/-   ##
=======================================
  Coverage   92.09%   92.09%           
=======================================
  Files          34       34           
  Lines        4516     4516           
=======================================
  Hits         4159     4159           
  Misses        357      357           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c401ee...df73ebb. Read the comment docs.

Wrote a lot more about this feature here: https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/

Twitter: https://twitter.com/dracos/status/1481290103519592459

Is it possible to parse things like --ext-{plugin}-{arg} VALUE ?

The challenge here is avoiding clashes. What if a plugin adds an option that I later want to use for a new Datasette core feature? Or what if two plugins define the same option?

Maybe the solution is to make them use namespaces defined by the plugin name.

How about this:

datasette t.db --plugin-setting datasette-tiddlywiki db t.db

It's a bit verbose having an option that itself then takes THREE strings: plugin name, setting name, setting value - but it would work.

In my example, path matching happens at the application layer (being the Django channels URLRouter). That might be a somewhat exotic solution that would normally be solved by a proxy like Apache or Nginx. However, in my specific use case, this is a "feature" enabling me to do simple management of databases and metadata from within a Django admin app instance mapped in that same router.

Thank you for the quick reply! Just a quick observation, I am running this locally without a proxy, whereas your fly example seems to be running behind an apache proxy (if the name is accurate). Can it be that the apache proxy strips the prefix before it passes on the request to the daphne backend?

Looking closer at the code quoted above, it doesn't modify path or raw_path at all - ALL it does is add the route_path to the scope.

Oh wait! It looks like route_path is something I invented there.

Yup, I added it in https://github.com/simonw/datasette/commit/a63412152518581c6a3d4e142b937e27dabdbfdb - commit message says:

• new route_path key in request.scope storing the path that was used for routing with the base_url prefix stripped

So actually part of the mystery here is: why does the Fly hosted one NOT have that key?

The Daphne one has this key: 'route_path': '/-/asgi-scope',

Maybe Datasette's routing code needs to look out for that, if it's available, and use it to reconstruct the requested path?

The code in question is here: https://github.com/simonw/datasette/blob/8c401ee0f054de2f568c3a8302c9223555146407/datasette/app.py#L1143-L1149

I'm using the https://datasette.io/plugins/datasette-debug-asgi plugin to investigate.

On my laptop using Daphne I get this: http://127.0.0.1:8032/datasettes/-/asgi-scope

{'actor': None,
 'asgi': {'version': '3.0'},
 'client': ['127.0.0.1', 53767],
 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x1122aeef0>,
 'headers': [(b'host', b'127.0.0.1:8032'),
             (b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko'
              b'/20100101 Firefox/95.0'),
             (b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'avif,image/webp,*/*;q=0.8'),
             (b'accept-language', b'en-US,en;q=0.5'),
             (b'accept-encoding', b'gzip, deflate'),
             (b'dnt', b'1'),
             (b'connection', b'keep-alive'),
             (b'cookie', b'_ga=GA1.1.742283954.1628542653'),
             (b'upgrade-insecure-requests', b'1'),
             (b'sec-fetch-dest', b'document'),
             (b'sec-fetch-mode', b'navigate'),
             (b'sec-fetch-site', b'none'),
             (b'sec-fetch-user', b'?1')],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/datasettes/-/asgi-scope',
 'path_remaining': '',
 'query_string': b'',
 'raw_path': b'/datasettes/-/asgi-scope',
 'root_path': '',
 'route_path': '/-/asgi-scope',
 'scheme': 'http',
 'server': ['127.0.0.1', 8032],
 'type': 'http',
 'url_route': {'kwargs': {}}}

On the demo running on Fly (which I just redeployed with that plugin) I get this: https://datasette-apache-proxy-demo.fly.dev/prefix/-/asgi-scope

{'actor': None,
 'asgi': {'spec_version': '2.1', 'version': '3.0'},
 'client': ('86.109.12.167', 0),
 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x7f4c0413bca0>,
 'headers': [(b'host', b'datasette-apache-proxy-demo.fly.dev'),
             (b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko'
              b'/20100101 Firefox/95.0'),
             (b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'avif,image/webp,*/*;q=0.8'),
             (b'accept-language', b'en-US,en;q=0.5'),
             (b'accept-encoding', b'gzip, deflate, br'),
             (b'dnt', b'1'),
             (b'x-request-start', b't=1641950740651658'),
             (b'sec-fetch-dest', b'document'),
             (b'sec-fetch-mode', b'navigate'),
             (b'sec-fetch-site', b'none'),
             (b'sec-fetch-user', b'?1'),
             (b'fly-client-ip', b'24.5.172.176'),
             (b'x-forwarded-for',
              b'24.5.172.176, 213.188.193.173, 86.109.12.167'),
             (b'fly-forwarded-proto', b'https'),
             (b'x-forwarded-proto', b'https'),
             (b'fly-forwarded-ssl', b'on'),
             (b'x-forwarded-ssl', b'on'),
             (b'fly-forwarded-port', b'443'),
             (b'x-forwarded-port', b'443'),
             (b'fly-region', b'sjc'),
             (b'fly-request-id', b'01FS5Y805BX43HM94T8XW610KG'),
             (b'via', b'2 fly.io'),
             (b'fly-dispatch-start', b't=1641950740683198;instance=87f188a2'),
             (b'x-forwarded-host', b'datasette-apache-proxy-demo.fly.dev'),
             (b'x-forwarded-server', b'localhost'),
             (b'connection', b'Keep-Alive')],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/-/asgi-scope',
 'query_string': b'',
 'raw_path': b'/-/asgi-scope',
 'root_path': '',
 'scheme': 'https',
 'server': ('127.0.0.1', 8001),
 'type': 'http',
 'url_route': {'kwargs': {}}}

The version that works as 'raw_path': b'/-/asgi-scope' - the version that fails has 'raw_path': b'/datasettes/-/asgi-scope'.

Thanks for the steps to reproduce - I have your bug running on my laptop now.

I've been mostly testing this stuff using the hosted copy of Datasette here, which doesn't exhibit the bug: https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures?sql=select+sqlite_version%28%29

Something interesting definitely going on here!

Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table

Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table

sqlite-utils search has --limit already: https://sqlite-utils.datasette.io/en/latest/cli-reference.html#search

  --limit INTEGER        Number of rows to return - defaults to everything

Twitter thread about this: https://twitter.com/simonw/status/1481020195074293761

The new page of documentation: https://sqlite-utils.datasette.io/en/latest/cli-reference.html

Here's the cog code I used: https://github.com/simonw/sqlite-utils/blob/1d44b0cc2784c94aed1bcf350225cd86ee1aa7e5/docs/cli-reference.rst#L11-L76

I'll call the new page "CLI reference", for consistency with the API reference page here: https://sqlite-utils.datasette.io/en/stable/reference.html

thanks so much! always a pleasure to see how you work through these things

CLI and Python library improvements to help run ANALYZE after creating indexes or inserting rows, to gain better performance from the SQLite query planner when it runs against indexes.

Three new CLI commands: create-database, analyze and bulk.

• New sqlite-utils create-database command for creating new empty database files. (#348)
• New Python methods for running ANALYZE against a database, table or index: db.analyze() and table.analyze(), see Optimizing index usage with ANALYZE. (#366)
• New sqlite-utils analyze command for running ANALYZE using the CLI. (#379)
• The create-index, insert and update commands now have a new --analyze option for running ANALYZE after the command has completed. (#379)
• New sqlite-utils bulk command which can import records in the same way as sqlite-utils insert (from JSON, CSV or TSV) and use them to bulk execute a parametrized SQL query. (#375)
• The CLI tool can now also be run using python -m sqlite_utils. (#368)
• Using --fmt now implies --table, so you don't need to pass both options. (#374)
• The --convert function applied to rows can now modify the row in place. (#371)
• The insert-files command supports two new columns: stem and suffix. (#372)
• The --nl import option now ignores blank lines in the input. (#376)
• Fixed bug where streaming input to the insert command with --batch-size 1 would appear to only commit after several rows had been ingested, due to unnecessary input buffering. (#364)

Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-sql-in-bulk

I tested this like so:

% wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
% sqlite-utils create-database test.db
% sqlite-utils create-table test.db power_plants url text owner text
% sqlite-utils schema test.db 
CREATE TABLE [power_plants] (
   [url] TEXT,
   [owner] TEXT
);
% sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv
  [------------------------------------]    0%
  [###################################-]   99%
% sqlite-utils tables --counts test.db -t
table           count
------------  -------
power_plants    33643

Documentation: https://github.com/simonw/sqlite-utils/blob/f4ea0d32c0543373eefaa9b9f3911eb07549eecb/docs/cli.rst#executing-sql-in-bulk

Codecov Report

Merging #377 (f4ea0d3) into main (b6dad08) will decrease coverage by 0.01%.
The diff coverage is 96.00%.

@@            Coverage Diff             @@
##             main     #377      +/-   ##
==========================================
- Coverage   96.52%   96.50%   -0.02%     
==========================================
  Files           6        6              
  Lines        2330     2378      +48     
==========================================
+ Hits         2249     2295      +46     
- Misses         81       83       +2     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6dad08...f4ea0d3. Read the comment docs.

I decided to go with making this opt-in, mainly for consistency with the other places where I added this feature - see:
- #379
- #366

You can now run the following:

sqlite-utils create-index mydb.db mytable mycolumn --analyze

And ANALYZE will be run on the index once it has been created.

The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.

Having browsed the API reference I think the methods that would benefit from an analyze=True parameter are:

• db.create_index
• table.insert_all
• table.upsert_all
• table.delete_where

Documentation for those two new methods: https://sqlite-utils.datasette.io/en/latest/python-api.html#optimizing-index-usage-with-analyze

With respect to #365, I'm now thinking that having the ability to say "... and then run ANALYZE" could be useful for a bunch of Python methods. For example:

db["dogs"].insert_all(list_of_dogs, analyze=True)
db["dogs"].create_index(["name"], analyze=True)

I'm going to implement the Python library methods based on the prototype:

commit 650f97a08f29a688c530e5f6c9eedc9269ed7bdc
Author: Simon Willison <swillison@gmail.com>
Date:   Sat Jan 8 13:34:01 2022 -0800

    Initial prototype of .analyze(), refs #366

diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index dfc4723..1348b4a 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -923,6 +923,13 @@ class Database:
         "Run a SQLite ``VACUUM`` against the database."
         self.execute("VACUUM;")

+    def analyze(self, name=None):
+        "Run ``ANALYZE`` against the entire database or a named table or index."
+        sql = "ANALYZE"
+        if name is not None:
+            sql += " [{}]".format(name)
+        self.execute(sql)
+

 class Queryable:
     def exists(self) -> bool:
@@ -2902,6 +2909,10 @@ class Table(Queryable):
             )
         return self

+    def analyze(self):
+        "Run ANALYZE against this table"
+        self.db.analyze(self.name)
+
     def analyze_column(
         self, column: str, common_limit: int = 10, value_truncate=None, total_rows=None
     ) -> "ColumnDetails":

I'm going to implement this in a separate commit from this PR.

That did the trick.

I tested the prototype like this:

sqlite-utils blah.db 'create table blah (id integer primary key, name text)' 
echo 'id,name
1,Cleo
2,Chicken' > blah.csv
sqlite-utils bulk blah.db 'insert into blah (id, name) values (:id, :name)' blah.csv --csv

Bit nasty but it might work:

    def try_until(expected):
        tries = 0
        while True:
            rows = list(Database(db_path)["rows"].rows)
            if rows == expected:
                return
            tries += 1
            if tries > 10:
                assert False, "Expected {}, got {}".format(expected, rows)
            time.sleep(tries * 0.1)

    try_until([{"name": "Azi"}])
    proc.stdin.write(b'{"name": "Suna"}\n')
    proc.stdin.flush()
    try_until([{"name": "Azi"}, {"name": "Suna"}])

Urgh, tests are still failing intermittently - for example:

        time.sleep(0.4)
>       assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}]
E       AssertionError: assert [] == [{'name': 'Azi'}]
E         Right contains one more item: {'name': 'Azi'}
E         Full diff:
E         - [{'name': 'Azi'}]
E         + []

I'm going to change this code to keep on trying up to 10 seconds - that should get the tests to pass faster on most machines.

After a bunch of debugging with print() statements it's clear that the problem isn't with when things are committed or the size of the batches - it's that the data sent to standard input is all being processed in one go, not a line at a time.

I think that's because it is being buffered by this: https://github.com/simonw/sqlite-utils/blob/d2a79d200f9071a86027365fa2a576865b71064f/sqlite_utils/cli.py#L759-L770

The buffering is there so that we can sniff the first few bytes to detect if it's a CSV file - added in 99ff0a288c08ec2071139c6031eb880fa9c95310 for #230. So maybe for non-CSV inputs we should disable buffering?

I think this test is right:

def test_insert_streaming_batch_size_1(db_path):
    # https://github.com/simonw/sqlite-utils/issues/364
    # Streaming with --batch-size 1 should commit on each record
    # Can't use CliRunner().invoke() here bacuse we need to
    # run assertions in between writing to process stdin
    proc = subprocess.Popen(
        [
            sys.executable,
            "-m",
            "sqlite_utils",
            "insert",
            db_path,
            "rows",
            "-",
            "--nl",
            "--batch-size",
            "1",
        ],
        stdin=subprocess.PIPE,
    )
    proc.stdin.write(b'{"name": "Azi"}')
    proc.stdin.flush()
    assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}]
    proc.stdin.write(b'{"name": "Suna"}')
    proc.stdin.flush()
    assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}, {"name": "Suna"}]
    proc.stdin.close()
    proc.wait()

Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#creating-an-empty-database

I'm not going to implement --page-size unless someone specifically requests it - I don't like having features that I've never needed to use myself.

The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added return, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic Error: Could not compile code message.

This seems to work:

def _compile_code(code, imports, variable="value"):
    locals = {}
    globals = {"r": recipes, "recipes": recipes}
    # If user defined a convert() function, return that
    try:
        exec(code, globals, locals)
        return locals["convert"]
    except (AttributeError, SyntaxError, NameError, KeyError, TypeError):
        pass

    # Try compiling their code as a function instead
    body_variants = [code]
    # If single line and no 'return', try adding the return
    if "\n" not in code and not code.strip().startswith("return "):
        body_variants.insert(0, "return {}".format(code))

    for variant in body_variants:
        new_code = ["def fn({}):".format(variable)]
        for line in variant.split("\n"):
            new_code.append("    {}".format(line))
        try:
            code_o = compile("\n".join(new_code), "<string>", "exec")
            break
        except SyntaxError:
            # Try another variant, e.g. for 'return row["column"] = 1'
            continue

    for import_ in imports:
        globals[import_.split(".")[0]] = __import__(import_)
    exec(code_o, globals, locals)
    return locals["fn"]

Here's the code in question: https://github.com/simonw/sqlite-utils/blob/b8c134059e89f0fa040b84fb7d0bda25b9a52759/sqlite_utils/utils.py#L288-L299

Tried this test:

    result = CliRunner().invoke(
        cli.cli,
        [
            "insert",
            db_path,
            "rows",
            "-",
            "--convert",
            'row["is_chicken"] = True',
        ],
        input='{"name": "Azi"}',
    )

And got this error:

E + where 1 = <Result SyntaxError('invalid syntax', ('<string>', 2, 30, ' return row["is_chicken"] = True\n'))>.exit_code

The code snippet compilation isn't currently compatible with this.

In addition to a unit test I manually tested all of the above, e.g.

% sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst
=======  ============  =======  =====  ======  ======  ======  =====
table    index_name    seqno    cid    name    desc    coll    key
=======  ============  =======  =====  ======  ======  ======  =====
=======  ============  =======  =====  ======  ======  ======  =====

Commands that support --fmt (via the @output_options decorator) are:

• tables
• views
• query
• memory
• search
• rows
• triggers
• indexes

Better:

if fmt:
    table = True

Updated TIL: https://til.simonwillison.net/python/cog-to-update-help-in-readme#user-content-cog-for-restructuredtext

Updated docs: https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output

Found an example of using cog in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21

.. [[[cog
    from coverage.sqldata import SCHEMA_VERSION
    print(".. code::")
    print()
    print(f"    SCHEMA_VERSION = {SCHEMA_VERSION}")
    print()
.. ]]]
.. code::

    SCHEMA_VERSION = 7

.. [[[end]]]

cat blah.csv | sqlite-utils bulk blah.db - \
   "insert into blah (:foo, :bar)" --csv

i don’t want to be such a partisan for analyze, but the query planner deciding not to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice.

<s>the original poster in that stack overflow doesn’t say there’s a performance regression </s>

@fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case.

@simonw any chance to take a look at this? If there is no interest, feel free to close the PR

Need to change if table: to if table or fmt: in a few places.

Pathlib says the stem of that would be dogs.and.cats.jpg - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix

It calls the last bit suffix - maybe I should use that instead of ext.

Also need to update relevant docs for that example.

Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828

To:

docs = (fn(doc) or doc for doc in docs)