html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/2143#issuecomment-1692210044,https://api.github.com/repos/simonw/datasette/issues/2143,1692210044,IC_kwDOBm6k_c5k3RN8,9599,2023-08-24T18:28:27Z,2023-08-24T18:28:27Z,OWNER,"Just spotted this: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L328-L332
https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L359-L360
Looks to me like that second bit of code doesn't yet handle `datasette.yml`
This code does though:
https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L333-L335
`parse_metadata()` is clearly a bad name for this function:
https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L980-L990
That ` @documented` decorator indicates that it's part of the documented API used by plugin authors: https://docs.datasette.io/en/1.0a4/internals.html#parse-metadata-content
So we should rename it to something better like `parse_json_or_yaml()` but keep `parse_metadata` as an undocumented alias for that to avoid any unnecessary plugin breaks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1692182910,https://api.github.com/repos/simonw/datasette/issues/2143,1692182910,IC_kwDOBm6k_c5k3Kl-,9599,2023-08-24T18:06:57Z,2023-08-24T18:08:17Z,OWNER,"The other thing that could work is something like this:
```bash
export AUTH_TOKENS_DB=""tokens""
datasette \
-s settings.sql_time_limit_ms 1000 \
-s plugins.datasette-auth-tokens.manage_tokens true \
-e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB
```
So `-e` is an alternative version of `-s` which reads from the named environment variable instead of having the value provided directly as the second value in the pair.
I quite like this, because it could replace the really ugly `$ENV` pattern we have in plugin configuration at the moment: https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values
```yaml
plugins:
datasette-auth-github:
client_secret:
$env: GITHUB_CLIENT_SECRET
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1692180683,https://api.github.com/repos/simonw/datasette/issues/2143,1692180683,IC_kwDOBm6k_c5k3KDL,9599,2023-08-24T18:05:17Z,2023-08-24T18:05:17Z,OWNER,"That's a really good call, thanks @rclement - environment variable configuration totally makes sense here.
Need to figure out the right syntax for that. Something like this perhaps:
```bash
DATASETTE_CONFIG_PLUGINS='{""datasette-ripgrep"": ...}'
```
Hard to know how to make this nestable though. I considered this:
```bash
DATASETTE_CONFIG_PLUGINS_DATASETTE_RIPGREP_PATH='/path/to/code/'
```
But that doesn't work, because how does the processing code know that it should split on `_` for most of the tokens but NOT split `DATASETTE_RIPGREP`, instead treating that as `datasette-ripgrep`?
I checked and `-` is not a valid character in an environment variable, at least in zsh on macOS:
```
% export FOO_BAR-BAZ=1
export: not valid in this context: FOO_BAR-BAZ
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690800119,https://api.github.com/repos/simonw/datasette/issues/2143,1690800119,IC_kwDOBm6k_c5kx4_3,9599,2023-08-24T00:10:32Z,2023-08-24T00:39:00Z,OWNER,"Something notable about this design is that, because the values in the key-value pairs are treated as JSON first and then strings only if they don't parse cleanly as JSON, it's possible to represent any structure (including nesting structures) using this syntax. You can do things like this if you need to (settings for an imaginary plugin):
```bash
datasette data.db \
-s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}'
```
Which would be equivalent to:
```yaml
plugins:
datasette-complex-plugin:
configs:
foo:
- 1
- 2
- 3
bar: baz
```
This is a bit different from a previous attempt I made at the same problem: https://github.com/simonw/json-flatten - that used syntax like `foo.bar.[0]$int = 1` to specify an integer as the first item of an array, which is much more complex.
That previous design was meant to support round-trips, so you could take any nested JSON object and turn it into an HTMl form or query string where every value can have its own form field, then turn the result back again.
For the `datasette -s key value` feature we don't need round-tripping with individual values each editable on their own, so we can go with something much simpler.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690800641,https://api.github.com/repos/simonw/datasette/issues/2143,1690800641,IC_kwDOBm6k_c5kx5IB,9599,2023-08-24T00:11:16Z,2023-08-24T00:11:16Z,OWNER,"> @simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.
That's a neat example thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514,https://api.github.com/repos/simonw/datasette/issues/2143,1690792514,IC_kwDOBm6k_c5kx3JC,9599,2023-08-24T00:00:16Z,2023-08-24T00:02:55Z,OWNER,"I've been thinking about what it might look like to allow command-line arguments to be used to define _any_ of the configuration options in `datasette.yml`, as alternative and more convenient syntax.
Here's what I've come up with:
```
datasette \
-s settings.sql_time_limit_ms 1000 \
-s plugins.datasette-auth-tokens.manage_tokens true \
-s plugins.datasette-auth-tokens.manage_tokens_database tokens \
-s plugins.datasette-ripgrep.path ""/home/simon/code-to-search"" \
-s databases.mydatabase.tables.example_table.sort created \
mydatabase.db tokens.db
```
Which would be equivalent to `datasette.yml` containing this:
```yaml
plugins:
datasette-auth-tokens:
manage_tokens: true
manage_tokens_database: tokens
datasette-ripgrep:
path: /home/simon/code-to-search
databases:
mydatabase:
tables:
example_table:
sort: created
settings:
sql_time_limit_ms: 1000
```
Here's a prototype implementation of this:
```python
import json
from typing import Any, List, Tuple
def _handle_pair(key: str, value: str) -> dict:
""""""
Turn a key-value pair into a nested dictionary.
foo, bar => {'foo': 'bar'}
foo.bar, baz => {'foo': {'bar': 'baz'}}
foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}}
foo.bar, ""baz"" => {'foo': {'bar': 'baz'}}
foo.bar, '{""baz"": ""qux""}' => {'foo': {'bar': ""{'baz': 'qux'}""}}
""""""
try:
value = json.loads(value)
except json.JSONDecodeError:
# If it doesn't parse as JSON, treat it as a string
pass
keys = key.split('.')
result = current_dict = {}
for k in keys[:-1]:
current_dict[k] = {}
current_dict = current_dict[k]
current_dict[keys[-1]] = value
return result
def _combine(base: dict, update: dict) -> dict:
""""""
Recursively merge two dictionaries.
""""""
for key, value in update.items():
if isinstance(value, dict) and key in base and isinstance(base[key], dict):
base[key] = _combine(base[key], value)
else:
base[key] = value
return base
def handle_pairs(pairs: List[Tuple[str, Any]]) -> dict:
""""""
Parse a list of key-value pairs into a nested dictionary.
""""""
result = {}
for key, value in pairs:
parsed_pair = _handle_pair(key, value)
result = _combine(result, parsed_pair)
return result
```
Exercised like this:
```python
print(json.dumps(handle_pairs([
(""settings.sql_time_limit_ms"", ""1000""),
(""plugins.datasette-auth-tokens.manage_tokens"", ""true""),
(""plugins.datasette-auth-tokens.manage_tokens_database"", ""tokens""),
(""plugins.datasette-ripgrep.path"", ""/home/simon/code-to-search""),
(""databases.mydatabase.tables.example_table.sort"", ""created""),
]), indent=4))
```
Output:
```json
{
""settings"": {
""sql_time_limit_ms"": 1000
},
""plugins"": {
""datasette-auth-tokens"": {
""manage_tokens"": true,
""manage_tokens_database"": ""tokens""
},
""datasette-ripgrep"": {
""path"": ""/home/simon/code-to-search""
}
},
""databases"": {
""mydatabase"": {
""tables"": {
""example_table"": {
""sort"": ""created""
}
}
}
}
}
```
Note that `-s` isn't currently an option for `datasette serve`.
`--setting key value` IS an existing option, but it isn't completely compatible with this because it maps directly just to settings.
Although... we could keep compatibility by saying that if you call `--setting known_setting value` and that `known_setting` is in this list then we treat it as if you said `-s settings.known_setting value` instead:
https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/app.py#L114-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,