Seeing as this area of the code has produced so many bugs in the past, I think part of the fix may be to write comprehensive documentation about how routing works for the internals documentation. Doing so might help me figure this bug out!

Most of these are already written for these two alpha releases: - https://github.com/simonw/datasette/releases/tag/0.60a0 - https://github.com/simonw/datasette/releases/tag/0.60a1

This works now: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable

Challenge: explain the difference between view as in SQL view, and view as in the code that serves TableView / DatabaseView etc.

See also: - #1412

Going with this for the copy:

Datasette Desktop is a packaged Mac application which bundles Datasette together with Python and allows you to install and run Datasette directly on your laptop. This is the best option for local installation if you are not comfortable using the command line.

https://docs.datasette.io/en/latest/cli-reference.html

I went with a simpler pattern that sqlite-utils because Datasette has a lot less commands: https://github.com/simonw/datasette/blob/4b23f01f3e668c8f2a2f1a294be49f49b4073969/docs/cli-reference.rst#L9-L35

The implementation can be almost exactly the same as this: https://github.com/simonw/sqlite-utils/blame/74586d3cb26fa3cc3412721985ecdc1864c2a31d/docs/cli-reference.rst#L11-L76

I need to do something extra to ensure the output of datasette --help-settings is shown too.

You can already run datasette --help-settings to see detailed help on available settings. Maybe datasette --help-plugin-settings could do the same thing for plugin settings?

Or I could even have available plugin settings show up as a list at the bottom of the datasette --help-settings output - which currently looks like this:

% datasette --help-settings Settings: default_page_size Default page size for the table view (default=100) max_returned_rows Maximum rows that can be returned from a table or custom query (default=1000) num_sql_threads Number of threads in the thread pool for executing SQLite queries (default=3) sql_time_limit_ms Time limit for a SQL query in milliseconds (default=1000) default_facet_size Number of values to return for requested facets (default=30) facet_time_limit_ms Time limit for calculating a requested facet (default=200) facet_suggest_time_limit_ms Time limit for calculating a suggested facet (default=50) hash_urls Include DB file contents hash in URLs, for far- future caching (default=False) allow_facet Allow users to specify columns to facet using ?_facet= parameter (default=True) allow_download Allow users to download the original SQLite database files (default=True) suggest_facets Calculate and display suggested facets (default=True) default_cache_ttl Default HTTP cache TTL (used in Cache-Control: max-age= header) (default=5) default_cache_ttl_hashed Default HTTP cache TTL for hashed URL pages (default=31536000) cache_size_kb SQLite cache size in KB (0 == use SQLite default) (default=0) allow_csv_stream Allow .csv?_stream=1 to download all rows (ignoring max_returned_rows) (default=True) max_csv_mb Maximum size allowed for CSV export in MB - set 0 to disable this limit (default=100) truncate_cells_html Truncate cells longer than this in HTML table view - set 0 to disable (default=2048) force_https_urls Force URLs in API output to always use https:// protocol (default=False) template_debug Allow display of template debug information with ?_context=1 (default=False) trace_debug Allow display of SQL trace debug information with ?_trace=1 (default=False) base_url Datasette URLs should use this base path (default=/)

Another option: if I make plugin settings a higher level concept in Datasette than they are at the moment, I could allow them to be set either using --options OR using the existing metadata.yml/json mechanism. https://docs.datasette.io/en/stable/plugins.html#plugin-configuration

I want to make changes to that anyway, because I'm increasingly uncomfortable with plugin settings ending up in the "metadata" mechanism.

I try to stick pretty closely to what Click supports, and Click likes you to define options explicitly so that it can display them in the output of --help.

But... that makes me think that actually showing these options in --help is likely a better idea. My --plugin-setting concept would have help that looks something like this: % datasette serve --help ... --plugin-setting <TEXT TEXT TEXT>... Setting for a specified plugin. That's not great help!

The alternative would be to allow plugins to register their extra options with the command - which would mean the help output could look like this instead: % datasette serve --help ... --tiddlywiki-db <TEXT> Name of database to use for datasette-tiddlywiki This feels like a good argument to me for plugins to explicitly register their settings. I'm not sure if I should enforce the name-of-plugin- prefix, or if I should allow plugins to define any setting they like. I'm still nervous about plugins over-riding existing or future default options to that command.

One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.

Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (gis.py or something) just so it's easier to find later.

Some proposals I'd add to sqlite-utils:

Some version of this, from geojson-to-sqlite:

python def init_spatialite(db, lib): db.conn.enable_load_extension(True) db.conn.load_extension(lib) # Initialize SpatiaLite if not yet initialized if "spatial_ref_sys" in db.table_names(): return db.conn.execute("select InitSpatialMetadata(1)")

Also a function for creating a spatial index:

python db.conn.execute("select CreateSpatialIndex(?, ?)", [table, "geometry"])

I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like:

sh sqlite-utils spatial-index spatial.db table-name column-name

Came here to add this. I might pick it up.

Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.

Codecov Report

Merging #1593 (df73ebb) into main (8c401ee) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #1593 +/-

======================================= Coverage 92.09% 92.09%
======================================= Files 34 34
Lines 4516 4516
======================================= Hits 4159 4159
Misses 357 357
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8c401ee...df73ebb. Read the comment docs.

Wrote a lot more about this feature here: https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/