issue_comments

Just spotted this: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L328-L332

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L359-L360

Looks to me like that second bit of code doesn't yet handle datasette.yml

This code does though:

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L333-L335

parse_metadata() is clearly a bad name for this function:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/init.py#L980-L990

That @documented decorator indicates that it's part of the documented API used by plugin authors: https://docs.datasette.io/en/1.0a4/internals.html#parse-metadata-content

So we should rename it to something better like parse_json_or_yaml() but keep parse_metadata as an undocumented alias for that to avoid any unnecessary plugin breaks.

The other thing that could work is something like this: bash export AUTH_TOKENS_DB="tokens" datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB So -e is an alternative version of -s which reads from the named environment variable instead of having the value provided directly as the second value in the pair.

I quite like this, because it could replace the really ugly $ENV pattern we have in plugin configuration at the moment: https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values yaml plugins: datasette-auth-github: client_secret: $env: GITHUB_CLIENT_SECRET

That's a really good call, thanks @rclement - environment variable configuration totally makes sense here.

Need to figure out the right syntax for that. Something like this perhaps:

bash DATASETTE_CONFIG_PLUGINS='{"datasette-ripgrep": ...}' Hard to know how to make this nestable though. I considered this: bash DATASETTE_CONFIG_PLUGINS_DATASETTE_RIPGREP_PATH='/path/to/code/' But that doesn't work, because how does the processing code know that it should split on _ for most of the tokens but NOT split DATASETTE_RIPGREP, instead treating that as datasette-ripgrep?

I checked and - is not a valid character in an environment variable, at least in zsh on macOS: % export FOO_BAR-BAZ=1 export: not valid in this context: FOO_BAR-BAZ

If I may, the "path-like" configuration is great but one thing that would be even greater: allowing the same configuration to be provided using environment variables.

For instance:

datasette -s plugins.datasette-complex-plugin.configs '{"foo": [1,2,3], "bar": "baz"}'

could also be provided using:

export DS_PLUGINS_DATASETTE-COMPLEX-PLUGIN_CONFIGS='{"foo": [1,2,3], "bar": "baz"}' datasette

(I do not like mixing - and _ in env vars but I do not have a best easily reversible example at the moment)

FYI, you could take some inspiration from another great open source data project, Metabase: https://www.metabase.com/docs/latest/configuring-metabase/config-file https://www.metabase.com/docs/latest/configuring-metabase/environment-variables

Something notable about this design is that, because the values in the key-value pairs are treated as JSON first and then strings only if they don't parse cleanly as JSON, it's possible to represent any structure (including nesting structures) using this syntax. You can do things like this if you need to (settings for an imaginary plugin):

bash datasette data.db \ -s plugins.datasette-complex-plugin.configs '{"foo": [1,2,3], "bar": "baz"}' Which would be equivalent to: yaml plugins: datasette-complex-plugin: configs: foo: - 1 - 2 - 3 bar: baz This is a bit different from a previous attempt I made at the same problem: https://github.com/simonw/json-flatten - that used syntax like foo.bar.[0]$int = 1 to specify an integer as the first item of an array, which is much more complex.

That previous design was meant to support round-trips, so you could take any nested JSON object and turn it into an HTMl form or query string where every value can have its own form field, then turn the result back again.

For the datasette -s key value feature we don't need round-tripping with individual values each editable on their own, so we can go with something much simpler.

@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.

That's a neat example thanks!

@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.

I've been thinking about what it might look like to allow command-line arguments to be used to define any of the configuration options in datasette.yml, as alternative and more convenient syntax.

Here's what I've come up with: datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -s plugins.datasette-auth-tokens.manage_tokens_database tokens \ -s plugins.datasette-ripgrep.path "/home/simon/code-to-search" \ -s databases.mydatabase.tables.example_table.sort created \ mydatabase.db tokens.db Which would be equivalent to datasette.yml containing this: yaml plugins: datasette-auth-tokens: manage_tokens: true manage_tokens_database: tokens datasette-ripgrep: path: /home/simon/code-to-search databases: mydatabase: tables: example_table: sort: created settings: sql_time_limit_ms: 1000 Here's a prototype implementation of this: ```python import json from typing import Any, List, Tuple

def _handle_pair(key: str, value: str) -> dict: """ Turn a key-value pair into a nested dictionary. foo, bar => {'foo': 'bar'} foo.bar, baz => {'foo': {'bar': 'baz'}} foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}} foo.bar, "baz" => {'foo': {'bar': 'baz'}} foo.bar, '{"baz": "qux"}' => {'foo': {'bar': "{'baz': 'qux'}"}} """ try: value = json.loads(value) except json.JSONDecodeError: # If it doesn't parse as JSON, treat it as a string pass

keys = key.split('.')
result = current_dict = {}

for k in keys[:-1]:
    current_dict[k] = {}
    current_dict = current_dict[k]

current_dict[keys[-1]] = value
return result

def _combine(base: dict, update: dict) -> dict: """ Recursively merge two dictionaries. """ for key, value in update.items(): if isinstance(value, dict) and key in base and isinstance(base[key], dict): base[key] = _combine(base[key], value) else: base[key] = value return base

def handle_pairs(pairs: List[Tuple[str, Any]]) -> dict: """ Parse a list of key-value pairs into a nested dictionary. """ result = {} for key, value in pairs: parsed_pair = _handle_pair(key, value) result = _combine(result, parsed_pair) return result Exercised like this:python print(json.dumps(handle_pairs([ ("settings.sql_time_limit_ms", "1000"), ("plugins.datasette-auth-tokens.manage_tokens", "true"), ("plugins.datasette-auth-tokens.manage_tokens_database", "tokens"), ("plugins.datasette-ripgrep.path", "/home/simon/code-to-search"), ("databases.mydatabase.tables.example_table.sort", "created"), ]), indent=4)) Output:json { "settings": { "sql_time_limit_ms": 1000 }, "plugins": { "datasette-auth-tokens": { "manage_tokens": true, "manage_tokens_database": "tokens" }, "datasette-ripgrep": { "path": "/home/simon/code-to-search" } }, "databases": { "mydatabase": { "tables": { "example_table": { "sort": "created" } } } } } `` Note that-sisn't currently an option fordatasette serve`.

--setting key value IS an existing option, but it isn't completely compatible with this because it maps directly just to settings.

Although... we could keep compatibility by saying that if you call --setting known_setting value and that known_setting is in this list then we treat it as if you said -s settings.known_setting value instead:

https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/app.py#L114-L204