html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/950#issuecomment-1692494455,https://api.github.com/repos/simonw/datasette/issues/950,1692494455,IC_kwDOBm6k_c5k4Wp3,9599,2023-08-24T22:26:08Z,2023-08-24T22:26:08Z,OWNER,"Closing this issue in favour of this one:
- #2157 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,
https://github.com/simonw/datasette/issues/2157#issuecomment-1692465763,https://api.github.com/repos/simonw/datasette/issues/2157,1692465763,IC_kwDOBm6k_c5k4Ppj,9599,2023-08-24T21:54:29Z,2023-08-24T21:54:29Z,OWNER,"But yes, I'm a big +1 on this whole plan.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865869205,
https://github.com/simonw/datasette/issues/2157#issuecomment-1692465334,https://api.github.com/repos/simonw/datasette/issues/2157,1692465334,IC_kwDOBm6k_c5k4Pi2,9599,2023-08-24T21:54:09Z,2023-08-24T21:54:09Z,OWNER,"We discussed this in-person this morning and these notes reflect what we talked about perfectly.
I've had so many bugs with plugins that I've written myself that have forgotten to special-case the `_internal` database when looping through `datasette.databases.keys()` - removing it from there entirely would help a lot.
Just one tiny disagreement: for `datasette-comments` I think having it store things in `_internal` could be an option, but in most cases I expect users to chose NOT to do that - because being able to join against those tables for more advanced queries is going to be super useful.
Show me all rows in `foia_requests` with at least one associated comment in `datasette_comments.comments` kind of tihng.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865869205,
https://github.com/simonw/datasette/issues/1241#issuecomment-1692322342,https://api.github.com/repos/simonw/datasette/issues/1241,1692322342,IC_kwDOBm6k_c5k3som,52261150,2023-08-24T19:56:15Z,2023-08-24T20:09:52Z,NONE,"Something to think about, but I hate how long the url is when sharing a custom SQL query. Would it be possible to hash the query and state of a page instead so the url is more manageable? The mapping from hash to query would have to be stored in order to recover/lookup the page after sharing.
It's not uncommon to have things like this currently:
```https://global-power-plants.datasettes.com/global-power-plants?sql=select+rowid%2C+country%2C+country_long%2C+name%2C+gppd_idnr%2C+capacity_mw%2C+latitude%2C+longitude%2C+primary_fuel%2C+other_fuel1%2C+other_fuel2%2C+other_fuel3%2C+commissioning_year%2C+owner%2C+source%2C+url%2C+geolocation_source%2C+wepp_id%2C+year_of_capacity_data%2C+generation_gwh_2013%2C+generation_gwh_2014%2C+generation_gwh_2015%2C+generation_gwh_2016%2C+generation_gwh_2017%2C+generation_gwh_2018%2C+generation_gwh_2019%2C+generation_data_source%2C+estimated_generation_gwh_2013%2C+estimated_generation_gwh_2014%2C+estimated_generation_gwh_2015%2C+estimated_generation_gwh_2016%2C+estimated_generation_gwh_2017%2C+estimated_generation_note_2013%2C+estimated_generation_note_2014%2C+estimated_generation_note_2015%2C+estimated_generation_note_2016%2C+estimated_generation_note_2017+from+%5Bglobal-power-plants%5D+order+by+rowid+limit+101```
I'm thinking a plugin like [https://datasette.io/plugins/datasette-query-files](https://datasette.io/plugins/datasette-query-files), but could be created and managed from the UI (with the right permissions).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,
https://github.com/simonw/datasette/issues/2143#issuecomment-1692210044,https://api.github.com/repos/simonw/datasette/issues/2143,1692210044,IC_kwDOBm6k_c5k3RN8,9599,2023-08-24T18:28:27Z,2023-08-24T18:28:27Z,OWNER,"Just spotted this: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L328-L332
https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L359-L360
Looks to me like that second bit of code doesn't yet handle `datasette.yml`
This code does though:
https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L333-L335
`parse_metadata()` is clearly a bad name for this function:
https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L980-L990
That ` @documented` decorator indicates that it's part of the documented API used by plugin authors: https://docs.datasette.io/en/1.0a4/internals.html#parse-metadata-content
So we should rename it to something better like `parse_json_or_yaml()` but keep `parse_metadata` as an undocumented alias for that to avoid any unnecessary plugin breaks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2156#issuecomment-1692206200,https://api.github.com/repos/simonw/datasette/issues/2156,1692206200,IC_kwDOBm6k_c5k3QR4,9599,2023-08-24T18:25:23Z,2023-08-24T18:25:23Z,OWNER,"Ran out of time for this, I'll look at the next step next week.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,
https://github.com/simonw/datasette/issues/2156#issuecomment-1692201647,https://api.github.com/repos/simonw/datasette/issues/2156,1692201647,IC_kwDOBm6k_c5k3PKv,9599,2023-08-24T18:21:53Z,2023-08-24T18:21:53Z,OWNER,"Oops, that was meant to be a PR. It's just a utility function though so it's safe to land already. I'll do a PR for the actual integration of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,
https://github.com/simonw/datasette/issues/2156#issuecomment-1692186522,https://api.github.com/repos/simonw/datasette/issues/2156,1692186522,IC_kwDOBm6k_c5k3Lea,9599,2023-08-24T18:10:04Z,2023-08-24T18:10:04Z,OWNER,I have an implementation in https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514 too - I'm going to land that as a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,
https://github.com/simonw/datasette/issues/2143#issuecomment-1692182910,https://api.github.com/repos/simonw/datasette/issues/2143,1692182910,IC_kwDOBm6k_c5k3Kl-,9599,2023-08-24T18:06:57Z,2023-08-24T18:08:17Z,OWNER,"The other thing that could work is something like this:
```bash
export AUTH_TOKENS_DB=""tokens""
datasette \
-s settings.sql_time_limit_ms 1000 \
-s plugins.datasette-auth-tokens.manage_tokens true \
-e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB
```
So `-e` is an alternative version of `-s` which reads from the named environment variable instead of having the value provided directly as the second value in the pair.
I quite like this, because it could replace the really ugly `$ENV` pattern we have in plugin configuration at the moment: https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values
```yaml
plugins:
datasette-auth-github:
client_secret:
$env: GITHUB_CLIENT_SECRET
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1692180683,https://api.github.com/repos/simonw/datasette/issues/2143,1692180683,IC_kwDOBm6k_c5k3KDL,9599,2023-08-24T18:05:17Z,2023-08-24T18:05:17Z,OWNER,"That's a really good call, thanks @rclement - environment variable configuration totally makes sense here.
Need to figure out the right syntax for that. Something like this perhaps:
```bash
DATASETTE_CONFIG_PLUGINS='{""datasette-ripgrep"": ...}'
```
Hard to know how to make this nestable though. I considered this:
```bash
DATASETTE_CONFIG_PLUGINS_DATASETTE_RIPGREP_PATH='/path/to/code/'
```
But that doesn't work, because how does the processing code know that it should split on `_` for most of the tokens but NOT split `DATASETTE_RIPGREP`, instead treating that as `datasette-ripgrep`?
I checked and `-` is not a valid character in an environment variable, at least in zsh on macOS:
```
% export FOO_BAR-BAZ=1
export: not valid in this context: FOO_BAR-BAZ
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/pull/2154#issuecomment-1691845306,https://api.github.com/repos/simonw/datasette/issues/2154,1691845306,IC_kwDOBm6k_c5k14K6,9599,2023-08-24T14:57:39Z,2023-08-24T14:57:39Z,OWNER,"Notes on manual testing so far - it looks like this might be working!
- https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691842259,https://api.github.com/repos/simonw/datasette/issues/2102,1691842259,IC_kwDOBm6k_c5k13bT,9599,2023-08-24T14:55:54Z,2023-08-24T14:55:54Z,OWNER,"So what's needed to finish this is:
- Tests that demonstrate that nothing is revealed that shouldn't be by tokens restricted in this way
- Similar tests for other permissions like `create-table` that check that they work (and don't also need `view-instance` etc).
- Documentation","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713,https://api.github.com/repos/simonw/datasette/issues/2102,1691824713,IC_kwDOBm6k_c5k1zJJ,9599,2023-08-24T14:45:49Z,2023-08-24T14:45:49Z,OWNER,"I tested this out against a Datasette Cloud instance. I created a restricted token and tested it like this:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
'https://$INSTANCE/-/actor.json' | jq
```
```json
{
""actor"": {
""id"": ""245"",
""token"": ""dsatok"",
""token_id"": 2,
""_r"": {
""r"": {
""data"": {
""all_stocks"": [
""vt""
]
}
}
}
}
}
```
It can access the `all_stocks` demo table:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
'https://$INSTANCE/data/all_stocks.json?_size=1' | jq
```
```json
{
""ok"": true,
""next"": ""1"",
""rows"": [
{
""rowid"": 1,
""Date"": ""2013-01-02"",
""Open"": 79.12,
""High"": 79.29,
""Low"": 77.38,
""Close"": 78.43,
""Volume"": 140124866,
""Name"": ""AAPL""
}
],
""truncated"": false
}
```
Accessing the database returns just information about that table, even though other tables exist:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
'https://$INSTANCE/data.json?_size=1'
```
```json
{
""database"": ""data"",
""private"": true,
""path"": ""/data"",
""size"": 3796992,
""tables"": [
{
""name"": ""all_stocks"",
""columns"": [
""Date"",
""Open"",
""High"",
""Low"",
""Close"",
""Volume"",
""Name""
],
""primary_keys"": [],
""count"": 8813,
""hidden"": false,
""fts_table"": null,
""foreign_keys"": {
""incoming"": [],
""outgoing"": []
},
""private"": true
}
],
""hidden_count"": 0,
""views"": [],
""queries"": [],
""allow_execute_sql"": false,
""table_columns"": {}
}
```
And hitting the top-level `/.json` thing does the same - it reveals that table but not any of the other tables or databases:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
'https://$INSTANCE/.json?_size=1'
```
```json
{
""data"": {
""name"": ""data"",
""hash"": null,
""color"": ""8d777f"",
""path"": ""/data"",
""tables_and_views_truncated"": [
{
""name"": ""all_stocks"",
""columns"": [
""Date"",
""Open"",
""High"",
""Low"",
""Close"",
""Volume"",
""Name""
],
""primary_keys"": [],
""count"": null,
""hidden"": false,
""fts_table"": null,
""num_relationships_for_sorting"": 0,
""private"": false
}
],
""tables_and_views_more"": false,
""tables_count"": 1,
""table_rows_sum"": 0,
""show_table_row_counts"": false,
""hidden_table_rows_sum"": 0,
""hidden_tables_count"": 0,
""views_count"": 0,
""private"": false
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,
https://github.com/simonw/datasette/pull/2154#issuecomment-1691788400,https://api.github.com/repos/simonw/datasette/issues/2154,1691788400,IC_kwDOBm6k_c5k1qRw,9599,2023-08-24T14:25:56Z,2023-08-24T14:25:56Z,OWNER,"Can be tested with:
```bash
pip install https://github.com/simonw/datasette/archive/6d57a8c23043e99b27f7a2afbe58f4d58815fd51.zip
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,
https://github.com/simonw/datasette/issues/2153#issuecomment-1691779180,https://api.github.com/repos/simonw/datasette/issues/2153,1691779180,IC_kwDOBm6k_c5k1oBs,9599,2023-08-24T14:21:03Z,2023-08-24T14:21:03Z,OWNER,"`datasette serve` currently only has a `--get` - for this to be really useful it needs to grow `--post` and maybe other verbs too.
Which is a good argument for moving this functionality to `datasette client get ...` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,
https://github.com/simonw/datasette/pull/2152#issuecomment-1691767797,https://api.github.com/repos/simonw/datasette/issues/2152,1691767797,IC_kwDOBm6k_c5k1lP1,9599,2023-08-24T14:15:10Z,2023-08-24T14:15:10Z,OWNER,This is broken because Sphinx no longer supports Python 3.8.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865174661,
https://github.com/simonw/datasette/issues/2153#issuecomment-1691763427,https://api.github.com/repos/simonw/datasette/issues/2153,1691763427,IC_kwDOBm6k_c5k1kLj,9599,2023-08-24T14:12:43Z,2023-08-24T14:12:43Z,OWNER,Annoying that `datasette client ...` makes a great name both for a plugin that executes simulated queries against a local database (thanks to its similarity to the existing `datasette.client` Python API) but is also the ideal name for a command for running commands as a client of an external Datasette instance!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,
https://github.com/simonw/datasette/issues/2153#issuecomment-1691761685,https://api.github.com/repos/simonw/datasette/issues/2153,1691761685,IC_kwDOBm6k_c5k1jwV,9599,2023-08-24T14:11:41Z,2023-08-24T14:11:41Z,OWNER,"Another option: implement this as a plugin, providing a new command like `datasette get ...`
Or implement `datasette client get ...` as core commands or a plugin - except that clashes with the `datasette client` command that https://github.com/simonw/dclient adds (which is a tool for hitting remote Datasette instances, not running simulated queries through a local one).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691758168,https://api.github.com/repos/simonw/datasette/issues/2102,1691758168,IC_kwDOBm6k_c5k1i5Y,9599,2023-08-24T14:09:45Z,2023-08-24T14:09:45Z,OWNER,I'm going to implement this in a branch to make it easier to test out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,
https://github.com/simonw/datasette/issues/2153#issuecomment-1691753489,https://api.github.com/repos/simonw/datasette/issues/2153,1691753489,IC_kwDOBm6k_c5k1hwR,9599,2023-08-24T14:07:25Z,2023-08-24T14:09:16Z,OWNER,"Building that `""_r""` array is the main reason this would be useful, but it's also fiddly to get right.
`datasette create-token` has a design for that already: https://docs.datasette.io/en/1.0a4/authentication.html#datasette-create-token
```
datasette create-token root \
--secret mysecret \
--all view-instance \
--all view-table \
--database docs view-query \
--resource docs documents insert-row \
--resource docs documents update-row
```
Adding imitations of those options (excluding `--secret`, not needed here) to `datasette serve` would add a LOT of extra options, but it would also make it really convenient to attempt a request with a specific set of restrictions. Not sure if that would be worth the extra `--help` output or not.
I feel like the names would have to have a common prefix though. Maybe something like this:
```bash
datasette serve data.db --get `/data/mytable.json' \
--actor-id root \
--r-all view-instance \
--r-database data view-query \
--r-resource data documents update-row
```
Other options could be the longer `--restrict-all/--restrict-database/--restrict-resource`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,
https://github.com/simonw/datasette/issues/2143#issuecomment-1691094870,https://api.github.com/repos/simonw/datasette/issues/2143,1691094870,IC_kwDOBm6k_c5kzA9W,1238873,2023-08-24T06:43:40Z,2023-08-24T06:43:40Z,NONE,"If I may, the ""path-like"" configuration is great but one thing that would be even greater: allowing the same configuration to be provided using environment variables.
For instance:
```
datasette -s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}'
```
could also be provided using:
```
export DS_PLUGINS_DATASETTE-COMPLEX-PLUGIN_CONFIGS='{""foo"": [1,2,3], ""bar"": ""baz""}'
datasette
```
(I do not like mixing `-` and `_` in env vars but I do not have a best easily reversible example at the moment)
FYI, you could take some inspiration from another great open source data project, Metabase:
https://www.metabase.com/docs/latest/configuring-metabase/config-file
https://www.metabase.com/docs/latest/configuring-metabase/environment-variables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691045051,https://api.github.com/repos/simonw/datasette/issues/2102,1691045051,IC_kwDOBm6k_c5ky0y7,9599,2023-08-24T05:51:59Z,2023-08-24T05:51:59Z,OWNER,"With that fix in place, this works:
```bash
datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
""_r"": {
""r"": {
""fixtures"": {
""facetable"": [
""vt""
]
}
}
},
""a"": ""user""
}'
```
But this fails, because it's for a table not explicitly listed:
```bash
datasette fixtures.db --get '/fixtures/searchable.json' --actor '{
""_r"": {
""r"": {
""fixtures"": {
""facetable"": [
""vt""
]
}
}
},
""a"": ""user""
}'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691044283,https://api.github.com/repos/simonw/datasette/issues/2102,1691044283,IC_kwDOBm6k_c5ky0m7,9599,2023-08-24T05:51:02Z,2023-08-24T05:51:02Z,OWNER,"Also need to confirm that permissions like `insert-row`, `delete-row`, `create-table` etc don't also need special cases to ensure they get through the `view-instance` etc checks, if those exist for those actions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691043475,https://api.github.com/repos/simonw/datasette/issues/2102,1691043475,IC_kwDOBm6k_c5ky0aT,9599,2023-08-24T05:50:04Z,2023-08-24T05:50:04Z,OWNER,"On first test this seems to work!
```diff
diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py
index 63a66c3c..9303dac8 100644
--- a/datasette/default_permissions.py
+++ b/datasette/default_permissions.py
@@ -187,6 +187,30 @@ def permission_allowed_actor_restrictions(datasette, actor, action, resource):
return None
_r = actor.get(""_r"")
+ # Special case for view-instance: it's allowed if there are any view-database
+ # or view-table permissions defined
+ if action == ""view-instance"":
+ database_rules = _r.get(""d"") or {}
+ for rules in database_rules.values():
+ if ""vd"" in rules or ""view-database"" in rules:
+ return None
+ # Now check resources
+ resource_rules = _r.get(""r"") or {}
+ for _database, resources in resource_rules.items():
+ for rules in resources.values():
+ if ""vt"" in rules or ""view-table"" in rules:
+ return None
+
+ # Special case for view-database: it's allowed if there are any view-table permissions
+ # defined within that database
+ if action == ""view-database"":
+ database_name = resource
+ resource_rules = _r.get(""r"") or {}
+ resources_in_database = resource_rules.get(database_name) or {}
+ for rules in resources_in_database.values():
+ if ""vt"" in rules or ""view-table"" in rules:
+ return None
+
# Does this action have an abbreviation?
to_check = {action}
permission = datasette.permissions.get(action)
```
Needs a LOT of testing to make sure what it's doing is sensible though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691037971,https://api.github.com/repos/simonw/datasette/issues/2102,1691037971,IC_kwDOBm6k_c5kyzET,9599,2023-08-24T05:42:47Z,2023-08-24T05:42:47Z,OWNER,"I applied a fun trick to help test this out:
```diff
diff --git a/datasette/cli.py b/datasette/cli.py
index 58f89c1c..830f47ef 100644
--- a/datasette/cli.py
+++ b/datasette/cli.py
@@ -445,6 +445,10 @@ def uninstall(packages, yes):
""--token"",
help=""API token to send with --get requests"",
)
+@click.option(
+ ""--actor"",
+ help=""Actor to use for --get requests"",
+)
@click.option(""--version-note"", help=""Additional note to show on /-/versions"")
@click.option(""--help-settings"", is_flag=True, help=""Show available settings"")
@click.option(""--pdb"", is_flag=True, help=""Launch debugger on any errors"")
@@ -499,6 +503,7 @@ def serve(
root,
get,
token,
+ actor,
version_note,
help_settings,
pdb,
@@ -611,7 +616,10 @@ def serve(
headers = {}
if token:
headers[""Authorization""] = ""Bearer {}"".format(token)
- response = client.get(get, headers=headers)
+ cookies = {}
+ if actor:
+ cookies[""ds_actor""] = client.actor_cookie(json.loads(actor))
+ response = client.get(get, headers=headers, cookies=cookies)
click.echo(response.text)
exit_code = 0 if response.status == 200 else 1
sys.exit(exit_code)
```
This adds a `--actor` option to `datasette ... --get /path` which makes it easy to test an API endpoint using a fake actor with a set of `_r` restrictions.
With that in place I can try this, with a token that has view-instance and view-database and view-table:
```bash
datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
""_r"": {
""a"": [
""vi""
],
""d"": {
""fixtures"": [
""vd""
]
},
""r"": {
""fixtures"": {
""facetable"": [
""vt""
]
}
}
},
""a"": ""user""
}'
```
Or this, with a token that just has view-table but is missing the view-database and view-instance:
```bash
datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
""_r"": {
""r"": {
""fixtures"": {
""facetable"": [
""vt""
]
}
}
},
""a"": ""user""
}'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691036559,https://api.github.com/repos/simonw/datasette/issues/2102,1691036559,IC_kwDOBm6k_c5kyyuP,9599,2023-08-24T05:40:53Z,2023-08-24T05:40:53Z,OWNER,"There might be an easier way to solve this. Here's some permission checks that run when hitting `/fixtures/facetable.json`:
```
permission_allowed: action=view-table, resource=('fixtures', 'facetable'), actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}
File ""/datasette/views/table.py"", line 727, in table_view_traced
view_data = await table_view_data(
File ""/datasette/views/table.py"", line 875, in table_view_data
visible, private = await datasette.check_visibility(
File ""/datasette/app.py"", line 890, in check_visibility
await self.ensure_permissions(actor, permissions)
permission_allowed: action=view-database, resource=fixtures, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}
File ""/datasette/views/table.py"", line 727, in table_view_traced
view_data = await table_view_data(
File ""/datasette/views/table.py"", line 875, in table_view_data
visible, private = await datasette.check_visibility(
File ""/datasette/app.py"", line 890, in check_visibility
await self.ensure_permissions(actor, permissions)
permission_allowed: action=view-instance, resource=, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}
File ""/datasette/views/table.py"", line 727, in table_view_traced
view_data = await table_view_data(
File ""/datasette/views/table.py"", line 875, in table_view_data
visible, private = await datasette.check_visibility(
File ""/datasette/app.py"", line 890, in check_visibility
await self.ensure_permissions(actor, permissions)
```
That's with a token that has the view instance, view database and view table permissions required.
But... what if the restrictions logic said that if you have view-table you automatically also get view-database and view-instance?
Would that actually let people do anything they shouldn't be able to do? I don't think it would even let them see a list of tables that they weren't allowed to visit, so it might be OK.
I'll try that and see how it works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,
https://github.com/simonw/datasette/issues/2147#issuecomment-1690955706,https://api.github.com/repos/simonw/datasette/issues/2147,1690955706,IC_kwDOBm6k_c5kye-6,18899,2023-08-24T03:54:35Z,2023-08-24T03:54:35Z,NONE,"That's fair. The best idea I can think of is that if a plugin wanted to limit intensive queries, it could add LIMITs or something. A hook that gives you visibility of queries and maybe the option to reject felt a little more limited than the existing plugin hooks, so I was trying to think of what else one might want to do while looking at to-be-run queries.
But without a real motivating example, I see why you don't want to add that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690800119,https://api.github.com/repos/simonw/datasette/issues/2143,1690800119,IC_kwDOBm6k_c5kx4_3,9599,2023-08-24T00:10:32Z,2023-08-24T00:39:00Z,OWNER,"Something notable about this design is that, because the values in the key-value pairs are treated as JSON first and then strings only if they don't parse cleanly as JSON, it's possible to represent any structure (including nesting structures) using this syntax. You can do things like this if you need to (settings for an imaginary plugin):
```bash
datasette data.db \
-s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}'
```
Which would be equivalent to:
```yaml
plugins:
datasette-complex-plugin:
configs:
foo:
- 1
- 2
- 3
bar: baz
```
This is a bit different from a previous attempt I made at the same problem: https://github.com/simonw/json-flatten - that used syntax like `foo.bar.[0]$int = 1` to specify an integer as the first item of an array, which is much more complex.
That previous design was meant to support round-trips, so you could take any nested JSON object and turn it into an HTMl form or query string where every value can have its own form field, then turn the result back again.
For the `datasette -s key value` feature we don't need round-tripping with individual values each editable on their own, so we can go with something much simpler.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690800641,https://api.github.com/repos/simonw/datasette/issues/2143,1690800641,IC_kwDOBm6k_c5kx5IB,9599,2023-08-24T00:11:16Z,2023-08-24T00:11:16Z,OWNER,"> @simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.
That's a neat example thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690799608,https://api.github.com/repos/simonw/datasette/issues/2143,1690799608,IC_kwDOBm6k_c5kx434,77071,2023-08-24T00:09:47Z,2023-08-24T00:10:41Z,NONE,"@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514,https://api.github.com/repos/simonw/datasette/issues/2143,1690792514,IC_kwDOBm6k_c5kx3JC,9599,2023-08-24T00:00:16Z,2023-08-24T00:02:55Z,OWNER,"I've been thinking about what it might look like to allow command-line arguments to be used to define _any_ of the configuration options in `datasette.yml`, as alternative and more convenient syntax.
Here's what I've come up with:
```
datasette \
-s settings.sql_time_limit_ms 1000 \
-s plugins.datasette-auth-tokens.manage_tokens true \
-s plugins.datasette-auth-tokens.manage_tokens_database tokens \
-s plugins.datasette-ripgrep.path ""/home/simon/code-to-search"" \
-s databases.mydatabase.tables.example_table.sort created \
mydatabase.db tokens.db
```
Which would be equivalent to `datasette.yml` containing this:
```yaml
plugins:
datasette-auth-tokens:
manage_tokens: true
manage_tokens_database: tokens
datasette-ripgrep:
path: /home/simon/code-to-search
databases:
mydatabase:
tables:
example_table:
sort: created
settings:
sql_time_limit_ms: 1000
```
Here's a prototype implementation of this:
```python
import json
from typing import Any, List, Tuple
def _handle_pair(key: str, value: str) -> dict:
""""""
Turn a key-value pair into a nested dictionary.
foo, bar => {'foo': 'bar'}
foo.bar, baz => {'foo': {'bar': 'baz'}}
foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}}
foo.bar, ""baz"" => {'foo': {'bar': 'baz'}}
foo.bar, '{""baz"": ""qux""}' => {'foo': {'bar': ""{'baz': 'qux'}""}}
""""""
try:
value = json.loads(value)
except json.JSONDecodeError:
# If it doesn't parse as JSON, treat it as a string
pass
keys = key.split('.')
result = current_dict = {}
for k in keys[:-1]:
current_dict[k] = {}
current_dict = current_dict[k]
current_dict[keys[-1]] = value
return result
def _combine(base: dict, update: dict) -> dict:
""""""
Recursively merge two dictionaries.
""""""
for key, value in update.items():
if isinstance(value, dict) and key in base and isinstance(base[key], dict):
base[key] = _combine(base[key], value)
else:
base[key] = value
return base
def handle_pairs(pairs: List[Tuple[str, Any]]) -> dict:
""""""
Parse a list of key-value pairs into a nested dictionary.
""""""
result = {}
for key, value in pairs:
parsed_pair = _handle_pair(key, value)
result = _combine(result, parsed_pair)
return result
```
Exercised like this:
```python
print(json.dumps(handle_pairs([
(""settings.sql_time_limit_ms"", ""1000""),
(""plugins.datasette-auth-tokens.manage_tokens"", ""true""),
(""plugins.datasette-auth-tokens.manage_tokens_database"", ""tokens""),
(""plugins.datasette-ripgrep.path"", ""/home/simon/code-to-search""),
(""databases.mydatabase.tables.example_table.sort"", ""created""),
]), indent=4))
```
Output:
```json
{
""settings"": {
""sql_time_limit_ms"": 1000
},
""plugins"": {
""datasette-auth-tokens"": {
""manage_tokens"": true,
""manage_tokens_database"": ""tokens""
},
""datasette-ripgrep"": {
""path"": ""/home/simon/code-to-search""
}
},
""databases"": {
""mydatabase"": {
""tables"": {
""example_table"": {
""sort"": ""created""
}
}
}
}
}
```
Note that `-s` isn't currently an option for `datasette serve`.
`--setting key value` IS an existing option, but it isn't completely compatible with this because it maps directly just to settings.
Although... we could keep compatibility by saying that if you call `--setting known_setting value` and that `known_setting` is in this list then we treat it as if you said `-s settings.known_setting value` instead:
https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/app.py#L114-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,