This description includes a long HTML string
This demonstrates basic LIKE search permissions: debug-menu: id: '*' plugins: datasette-ripgrep: path: /usr/local/lib/python3.11/site-packages ``` I'm inclined to say we try to be a super-set of the existing `metadata.yml` format, at least where it makes sense to do so. That way the upgrade path is smooth for people. Also, I don't think the format itself is terrible - it's the name that's the big problem. In this example I've mixed in one extra concept: that `settings:` block with a bunch of settings in it. There are some things in there that look a little bit like metadata - the `title` and `description_html` fields. But _are they_ metadata? The title and description of the overall instance feels like it could be described as general configuration. The stuff for the `query` should live where the query itself is defined. Note that queries can be defined by a plugin hook too: https://docs.datasette.io/en/1.0a3/plugin_hooks.html#canned-queries-datasette-database-actor What do you think? Is this the right direction, or are you thinking there's a more radical redesign that would make sense here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1684485591,https://api.github.com/repos/simonw/datasette/issues/2143,1684485591,IC_kwDOBm6k_c5kZzXX,9599,simonw,2023-08-18T22:14:35Z,2023-08-18T22:14:35Z,OWNER,"Actually there is one thing that I'm not comfortable about with respect to the existing design: the way the database / tables stuff is nested. They assume that the user will attach the database to Datasette using a fixed name - `docs.db` or whatever. But what if we want to support users downloading databases from each other and attaching them to Datasette where those DBs might carry some of their own configuration? Moving metadata into the databases makes sense there, but what about database-specific settings like the default sort order for a table, or configured canned queries? Having those tied to the filename of the database itself feels unpleasant to me. But how else could we handle this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1684488526,https://api.github.com/repos/simonw/datasette/issues/2143,1684488526,IC_kwDOBm6k_c5kZ0FO,9599,simonw,2023-08-18T22:18:39Z,2023-08-18T22:18:39Z,OWNER,"> Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings. > Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings. I'm not a fan of that. I feel like software history is full of examples of projects that implemented configuration-as-code and then later regretted it - the most recent example is `setup.py` in Python turning into `pyproject.yaml`, but I feel like I've seen that pattern play out elsewhere too. I don't think having people dynamically generate JSON/YAML for their configuration is a big burden. I'd have to see some very compelling use-cases to convince me otherwise. That said, I do really like a bias towards settings that can be changed at runtime. Datasette has suffered a bit from some settings that can't be easily changed at runtime already - hence my gnarly https://github.com/simonw/datasette-remote-metadata plugin. For things like Datasette Cloud for example the more people can configure without rebooting their container the better! I don't think live reconfiguration at runtime is incompatible with JSON/YAML configuration though. Caddy is one of my favourite examples of software that can be entirely re-configured at runtime by POSTING a big blob of JSON to it: https://caddyserver.com/docs/quick-starts/api ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1684496274,https://api.github.com/repos/simonw/datasette/issues/2143,1684496274,IC_kwDOBm6k_c5kZ1-S,15178711,asg017,2023-08-18T22:30:45Z,2023-08-18T22:30:45Z,CONTRIBUTOR,"> That said, I do really like a bias towards settings that can be changed at runtime Does this include things like `--settings` values or plugin config? I can totally see being able to update metadata without restarting, but not sure if that would work well with `--setting`, plugin config, or auth/permissions stuff. Well it could work with `--setting` and auth/permissions, with a lot of core changes. But changing plugin config on the fly could be challenging, for plugin authors. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1685259985,https://api.github.com/repos/simonw/datasette/issues/2143,1685259985,IC_kwDOBm6k_c5kcwbR,11784304,dvizard,2023-08-20T11:27:21Z,2023-08-20T11:27:21Z,NONE,"To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate ""settings"" from ""configuration"" is, I believe, an uphill fight. Even separating db/table-scope settings from pure descriptive metadata is not always easy. Like, do canned queries belong to database metadata or to settings? Do I need two separate files for this? One pragmatic solution I used in a project is stacking yaml configuration files. Basically, have an arbitrary number of yaml or json settings files that you load in a specified order. Every file adds to the corresponding settings in the earlier-loaded file (if it already existed). I implemented this myself but found later that there is an existing Python ""cascading dict"" type of thing, I forget what it's called. There is a bit of a challenge deciding whether there is ""replacement"" or ""addition"" (I think I pragmatically ran `update` on the second level of the dict but better solutions are certainly possible). This way, one allows separation of settings into different blocks, while not imposing a specific idea of what belongs where that might not apply equally to all cases. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1685260244,https://api.github.com/repos/simonw/datasette/issues/2143,1685260244,IC_kwDOBm6k_c5kcwfU,11784304,dvizard,2023-08-20T11:29:00Z,2023-08-20T11:29:00Z,NONE,https://docs.python.org/3/library/collections.html#collections.ChainMap,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1685260624,https://api.github.com/repos/simonw/datasette/issues/2143,1685260624,IC_kwDOBm6k_c5kcwlQ,11784304,dvizard,2023-08-20T11:31:16Z,2023-08-20T11:31:16Z,NONE,https://pypi.org/project/deep-chainmap/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1685263948,https://api.github.com/repos/simonw/datasette/issues/2143,1685263948,IC_kwDOBm6k_c5kcxZM,11784304,dvizard,2023-08-20T11:50:10Z,2023-08-20T11:50:10Z,NONE,"This also makes it simple to separate out secrets. `datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml` settings.yaml ``` settings: default_page_size: 10 max_returned_rows: 3000 sql_time_limit_ms"": 8000 plugins: datasette-ripgrep: path: /usr/local/lib/python3.11/site-packages ``` secrets.yaml ``` plugins: datasette-auth-github: client_secret: SUCH_SECRET ``` db-docs.yaml ``` databases: docs: permissions: create-table: id: editor ``` db-fixtures.yaml ``` databases: fixtures: tables: no_primary_key: hidden: true queries: neighborhood_search: sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood; title: Search neighborhoods description_html: |-
This demonstrates basic LIKE search ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1690787394,https://api.github.com/repos/simonw/datasette/issues/2143,1690787394,IC_kwDOBm6k_c5kx15C,9599,simonw,2023-08-23T23:52:02Z,2023-08-23T23:52:02Z,OWNER,"> This also makes it simple to separate out secrets. > > `datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml` Having multiple configs that combine in that way is a really interesting direction. > To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate ""settings"" from ""configuration"" is, I believe, an uphill fight. I'm very keen on separating out the ""metadata"" - where metadata is the slimmest possible set of things, effectively the data license and the source and the column and table descriptions - from everything else, mainly because I want metadata to be able to travel with the data. One idea that's been discussed before is having an optional mechanism for storing metadata in the SQLite database file itself - potentially in a `_datasette_metadata` table. That way you could distribute a DB file and anyone who opened it in Datasette would also see the correct metadata about it. That's why I'm so keen on splitting out metadata from all of the other stuff - settings and plugin configuration and authentication rules. So really it becomes ""true metadata"" v.s. ""all of the other junk that's accumulated in metadata and `settings.json`"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514,https://api.github.com/repos/simonw/datasette/issues/2143,1690792514,IC_kwDOBm6k_c5kx3JC,9599,simonw,2023-08-24T00:00:16Z,2023-08-24T00:02:55Z,OWNER,"I've been thinking about what it might look like to allow command-line arguments to be used to define _any_ of the configuration options in `datasette.yml`, as alternative and more convenient syntax. Here's what I've come up with: ``` datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -s plugins.datasette-auth-tokens.manage_tokens_database tokens \ -s plugins.datasette-ripgrep.path ""/home/simon/code-to-search"" \ -s databases.mydatabase.tables.example_table.sort created \ mydatabase.db tokens.db ``` Which would be equivalent to `datasette.yml` containing this: ```yaml plugins: datasette-auth-tokens: manage_tokens: true manage_tokens_database: tokens datasette-ripgrep: path: /home/simon/code-to-search databases: mydatabase: tables: example_table: sort: created settings: sql_time_limit_ms: 1000 ``` Here's a prototype implementation of this: ```python import json from typing import Any, List, Tuple def _handle_pair(key: str, value: str) -> dict: """""" Turn a key-value pair into a nested dictionary. foo, bar => {'foo': 'bar'} foo.bar, baz => {'foo': {'bar': 'baz'}} foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}} foo.bar, ""baz"" => {'foo': {'bar': 'baz'}} foo.bar, '{""baz"": ""qux""}' => {'foo': {'bar': ""{'baz': 'qux'}""}} """""" try: value = json.loads(value) except json.JSONDecodeError: # If it doesn't parse as JSON, treat it as a string pass keys = key.split('.') result = current_dict = {} for k in keys[:-1]: current_dict[k] = {} current_dict = current_dict[k] current_dict[keys[-1]] = value return result def _combine(base: dict, update: dict) -> dict: """""" Recursively merge two dictionaries. """""" for key, value in update.items(): if isinstance(value, dict) and key in base and isinstance(base[key], dict): base[key] = _combine(base[key], value) else: base[key] = value return base def handle_pairs(pairs: List[Tuple[str, Any]]) -> dict: """""" Parse a list of key-value pairs into a nested dictionary. """""" result = {} for key, value in pairs: parsed_pair = _handle_pair(key, value) result = _combine(result, parsed_pair) return result ``` Exercised like this: ```python print(json.dumps(handle_pairs([ (""settings.sql_time_limit_ms"", ""1000""), (""plugins.datasette-auth-tokens.manage_tokens"", ""true""), (""plugins.datasette-auth-tokens.manage_tokens_database"", ""tokens""), (""plugins.datasette-ripgrep.path"", ""/home/simon/code-to-search""), (""databases.mydatabase.tables.example_table.sort"", ""created""), ]), indent=4)) ``` Output: ```json { ""settings"": { ""sql_time_limit_ms"": 1000 }, ""plugins"": { ""datasette-auth-tokens"": { ""manage_tokens"": true, ""manage_tokens_database"": ""tokens"" }, ""datasette-ripgrep"": { ""path"": ""/home/simon/code-to-search"" } }, ""databases"": { ""mydatabase"": { ""tables"": { ""example_table"": { ""sort"": ""created"" } } } } } ``` Note that `-s` isn't currently an option for `datasette serve`. `--setting key value` IS an existing option, but it isn't completely compatible with this because it maps directly just to settings. Although... we could keep compatibility by saying that if you call `--setting known_setting value` and that `known_setting` is in this list then we treat it as if you said `-s settings.known_setting value` instead: https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/app.py#L114-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1690799608,https://api.github.com/repos/simonw/datasette/issues/2143,1690799608,IC_kwDOBm6k_c5kx434,77071,pkulchenko,2023-08-24T00:09:47Z,2023-08-24T00:10:41Z,NONE,"@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1690800119,https://api.github.com/repos/simonw/datasette/issues/2143,1690800119,IC_kwDOBm6k_c5kx4_3,9599,simonw,2023-08-24T00:10:32Z,2023-08-24T00:39:00Z,OWNER,"Something notable about this design is that, because the values in the key-value pairs are treated as JSON first and then strings only if they don't parse cleanly as JSON, it's possible to represent any structure (including nesting structures) using this syntax. You can do things like this if you need to (settings for an imaginary plugin): ```bash datasette data.db \ -s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}' ``` Which would be equivalent to: ```yaml plugins: datasette-complex-plugin: configs: foo: - 1 - 2 - 3 bar: baz ``` This is a bit different from a previous attempt I made at the same problem: https://github.com/simonw/json-flatten - that used syntax like `foo.bar.[0]$int = 1` to specify an integer as the first item of an array, which is much more complex. That previous design was meant to support round-trips, so you could take any nested JSON object and turn it into an HTMl form or query string where every value can have its own form field, then turn the result back again. For the `datasette -s key value` feature we don't need round-tripping with individual values each editable on their own, so we can go with something much simpler.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1690800641,https://api.github.com/repos/simonw/datasette/issues/2143,1690800641,IC_kwDOBm6k_c5kx5IB,9599,simonw,2023-08-24T00:11:16Z,2023-08-24T00:11:16Z,OWNER,"> @simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case. That's a neat example thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1691094870,https://api.github.com/repos/simonw/datasette/issues/2143,1691094870,IC_kwDOBm6k_c5kzA9W,1238873,rclement,2023-08-24T06:43:40Z,2023-08-24T06:43:40Z,NONE,"If I may, the ""path-like"" configuration is great but one thing that would be even greater: allowing the same configuration to be provided using environment variables. For instance: ``` datasette -s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}' ``` could also be provided using: ``` export DS_PLUGINS_DATASETTE-COMPLEX-PLUGIN_CONFIGS='{""foo"": [1,2,3], ""bar"": ""baz""}' datasette ``` (I do not like mixing `-` and `_` in env vars but I do not have a best easily reversible example at the moment) FYI, you could take some inspiration from another great open source data project, Metabase: https://www.metabase.com/docs/latest/configuring-metabase/config-file https://www.metabase.com/docs/latest/configuring-metabase/environment-variables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1692180683,https://api.github.com/repos/simonw/datasette/issues/2143,1692180683,IC_kwDOBm6k_c5k3KDL,9599,simonw,2023-08-24T18:05:17Z,2023-08-24T18:05:17Z,OWNER,"That's a really good call, thanks @rclement - environment variable configuration totally makes sense here. Need to figure out the right syntax for that. Something like this perhaps: ```bash DATASETTE_CONFIG_PLUGINS='{""datasette-ripgrep"": ...}' ``` Hard to know how to make this nestable though. I considered this: ```bash DATASETTE_CONFIG_PLUGINS_DATASETTE_RIPGREP_PATH='/path/to/code/' ``` But that doesn't work, because how does the processing code know that it should split on `_` for most of the tokens but NOT split `DATASETTE_RIPGREP`, instead treating that as `datasette-ripgrep`? I checked and `-` is not a valid character in an environment variable, at least in zsh on macOS: ``` % export FOO_BAR-BAZ=1 export: not valid in this context: FOO_BAR-BAZ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1692182910,https://api.github.com/repos/simonw/datasette/issues/2143,1692182910,IC_kwDOBm6k_c5k3Kl-,9599,simonw,2023-08-24T18:06:57Z,2023-08-24T18:08:17Z,OWNER,"The other thing that could work is something like this: ```bash export AUTH_TOKENS_DB=""tokens"" datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB ``` So `-e` is an alternative version of `-s` which reads from the named environment variable instead of having the value provided directly as the second value in the pair. I quite like this, because it could replace the really ugly `$ENV` pattern we have in plugin configuration at the moment: https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values ```yaml plugins: datasette-auth-github: client_secret: $env: GITHUB_CLIENT_SECRET ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1692210044,https://api.github.com/repos/simonw/datasette/issues/2143,1692210044,IC_kwDOBm6k_c5k3RN8,9599,simonw,2023-08-24T18:28:27Z,2023-08-24T18:28:27Z,OWNER,"Just spotted this: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L328-L332 https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L359-L360 Looks to me like that second bit of code doesn't yet handle `datasette.yml` This code does though: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L333-L335 `parse_metadata()` is clearly a bad name for this function: https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L980-L990 That ` @documented` decorator indicates that it's part of the documented API used by plugin authors: https://docs.datasette.io/en/1.0a4/internals.html#parse-metadata-content So we should rename it to something better like `parse_json_or_yaml()` but keep `parse_metadata` as an undocumented alias for that to avoid any unnecessary plugin breaks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,