issue_comments

OK Here's the gameplan for this, which is closely tied to #2143 :

• We will add a new datasette.json/datasette.yaml configuration file to datasette, which combines settings/plugin config/permissions/canned queries into a new file format
• Metadata will NOT be a part of this file
• TOML support is not planned, but maybe we can create a separate issue for support TOML with JSON/YAML
• The settings.json file will be deprecated, and the --config arg will be brought back.
• Command line arguments can still be used to overwrite values (ex --setting will overwrite settings in datasette.yaml

The format of datasette.json will follow what Simon listed here: https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426

Here's the current implementation plan:

1. Add a new --config flag and port over "settings" into a new datasette.json config file, remove settings.json
2. Add top-level plugin config support to datasette.json
3. Figure out database/table structure of config datasette.json
4. Port over database/table level plugin config support datasette.json
5. Port over permissions/auth settings to datasette.json
6. Deprecate non-metadata values in metadata.json

Storing metadata in the database won't be required. I imagine there'll be many different ways to store metadata, including any possible datasette_metadata or sqlite-docs, or the older metadata.json way.

The next question will be how precedence should work - i'd imagine metadata.json > plugins > datasette_metadata > sqlite-docs

I'm not keen on requiring metadata to be within the database. I commonly have multiple DBs, from various sources, and having one config file to provide the metadata works out very well. I use Datasette with databases where I'm not the original source, needing to mutate them to add a metadata table or sqlite-docs makes me uncomfortable.

I agree, settings in the DB doesn't make sense but metadata does.

On the JSON v YAML v TOML issue I just spotted Caddy has a concept of config adapters which they use to resolve exactly that problem: https://caddyserver.com/docs/config-adapters

I do like also being able to set options using command line options though - for things like SQL time limits I'd much rather be able to throw on --setting sql_time_limit_ms 10000 than have to save a config file to disk.

So I'd want to support both. Which maybe means also having a way to set plugin options with CLI options. datasette publish kind of has that ability already:

datasette publish heroku my_database.db \ --name my-heroku-app-demo \ --install=datasette-auth-github \ --plugin-secret datasette-auth-github client_id your_client_id \ --plugin-secret datasette-auth-github client_secret your_client_secret

Maybe we can have a separate issue for revamping metadata.json? A datasette_metadata table or the sqlite-docs extension seem like two reasonable additions that we can work through. Storing metadata inside a SQLite database makes sense, but I don't think storing datasette.* style config (ex ports, settings, etc.) inside a SQLite DB makes sense, since it's very environment-dependent

I agree with not liking metadata.json stuff in a datasette.* config file. Editing description of a table/column in a file like datasette.* seems odd to me.

Though since plugin configuration currently lives in metadata.json, I think it should be removed from there and placed in datasette.*, at least for top-level config like datasette-auth-github's config. Keeping metadata.json strictly for documentation/licensing/column units makes sense to me, but anything plugin related should be in some config file, like datasette.*.

And ya, supporting both datasette.* and CLI flags makes a lot of sense to me. Any --setting flag should override anything in datasette.* for easier debugging, with possibly a warning message so people don't get confused. Same with --port and a port defined in datasette.*

I'm strongly in favour of combining settings, configuration and plugin configuration.

I'm not keen on mixing in metadata as well - that feels like a different concept to me, and I'm unhappy with how that's already had things like plugin settings leak into it.

I'm not yet sold on TOML - I actually find it less intuitive than YAML, surprisingly. They all have their warts I guess.

Datasette already has the ability to consume JSON or YAML for metadata - maybe it could grow TOML support too? That way users could have a datasette.json or datasette.yaml or datasette.toml file depending on their preference.

In terms of metadata: since that's means to be driven by a plugin hook anyway, maybe one of the potential sources of metadata is a metadata nested object in that datasette.* configuration file. Or you can have it in a separate metadata.json or bundled into the SQLite database or some other plugin-driven mechanism.