home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

33 rows where "created_at" is on date 2023-08-24 sorted by updated_at descending

✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, reactions, created_at (date), updated_at (date)

issue 10

  • API tokens with view-table but not view-database/view-instance cannot access the table 8
  • De-tangling Metadata before Datasette 1.0 8
  • Datasette --get --actor option 4
  • Cascade for restricted token view-table/view-database/view-instance operations 3
  • datasette -s/--setting option for setting nested configuration options 3
  • Bump the python-packages group with 3 updates 2
  • Proposal: Make the `_internal` database persistent, customizable, and hidden 2
  • Private/secret databases: database files that are only visible to plugins 1
  • Share button for copying current URL 1
  • Plugin hook for database queries that are run 1

user 6

  • simonw 27
  • codecov[bot] 2
  • jackowayed 1
  • pkulchenko 1
  • rclement 1
  • publicmatt 1

author_association 2

  • OWNER 27
  • NONE 6
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1691798722 https://github.com/simonw/datasette/pull/2154#issuecomment-1691798722 https://api.github.com/repos/simonw/datasette/issues/2154 IC_kwDOBm6k_c5k1szC codecov[bot] 22429695 2023-08-24T14:31:42Z 2023-08-29T16:15:12Z NONE

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.03% :tada:

Comparison is base (2e28258) 92.82% compared to head (3e49fd3) 92.85%. Report is 3 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2154 +/- ## ========================================== + Coverage 92.82% 92.85% +0.03% ========================================== Files 40 40 Lines 5948 6008 +60 ========================================== + Hits 5521 5579 +58 - Misses 427 429 +2 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/special.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `94.06% <ø> (-0.85%)` | :arrow_down: | | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.37% <100.00%> (+0.07%)` | :arrow_up: | | [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `97.48% <100.00%> (+0.57%)` | :arrow_up: | | [datasette/permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3Blcm1pc3Npb25zLnB5) | `100.00% <100.00%> (ø)` | | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2154/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Cascade for restricted token view-table/view-database/view-instance operations 1865281760  
1691710474 https://github.com/simonw/datasette/pull/2152#issuecomment-1691710474 https://api.github.com/repos/simonw/datasette/issues/2152 IC_kwDOBm6k_c5k1XQK codecov[bot] 22429695 2023-08-24T13:45:21Z 2023-08-25T13:18:34Z NONE

Codecov Report

Patch and project coverage have no change.

Comparison is base (527cec6) 92.76% compared to head (2e45686) 92.76%.

:exclamation: Current head 2e45686 differs from pull request most recent head 5dfa305. Consider uploading reports for the commit 5dfa305 to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2152 +/- ## ======================================= Coverage 92.76% 92.76% ======================================= Files 40 40 Lines 5943 5919 -24 ======================================= - Hits 5513 5491 -22 + Misses 430 428 -2 ``` [see 3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2152/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump the python-packages group with 3 updates 1865174661  
1692494455 https://github.com/simonw/datasette/issues/950#issuecomment-1692494455 https://api.github.com/repos/simonw/datasette/issues/950 IC_kwDOBm6k_c5k4Wp3 simonw 9599 2023-08-24T22:26:08Z 2023-08-24T22:26:08Z OWNER

Closing this issue in favour of this one: - #2157

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Private/secret databases: database files that are only visible to plugins 685806511  
1692465763 https://github.com/simonw/datasette/issues/2157#issuecomment-1692465763 https://api.github.com/repos/simonw/datasette/issues/2157 IC_kwDOBm6k_c5k4Ppj simonw 9599 2023-08-24T21:54:29Z 2023-08-24T21:54:29Z OWNER

But yes, I'm a big +1 on this whole plan.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proposal: Make the `_internal` database persistent, customizable, and hidden 1865869205  
1692465334 https://github.com/simonw/datasette/issues/2157#issuecomment-1692465334 https://api.github.com/repos/simonw/datasette/issues/2157 IC_kwDOBm6k_c5k4Pi2 simonw 9599 2023-08-24T21:54:09Z 2023-08-24T21:54:09Z OWNER

We discussed this in-person this morning and these notes reflect what we talked about perfectly.

I've had so many bugs with plugins that I've written myself that have forgotten to special-case the _internal database when looping through datasette.databases.keys() - removing it from there entirely would help a lot.

Just one tiny disagreement: for datasette-comments I think having it store things in _internal could be an option, but in most cases I expect users to chose NOT to do that - because being able to join against those tables for more advanced queries is going to be super useful.

Show me all rows in foia_requests with at least one associated comment in datasette_comments.comments kind of tihng.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Proposal: Make the `_internal` database persistent, customizable, and hidden 1865869205  
1692322342 https://github.com/simonw/datasette/issues/1241#issuecomment-1692322342 https://api.github.com/repos/simonw/datasette/issues/1241 IC_kwDOBm6k_c5k3som publicmatt 52261150 2023-08-24T19:56:15Z 2023-08-24T20:09:52Z NONE

Something to think about, but I hate how long the url is when sharing a custom SQL query. Would it be possible to hash the query and state of a page instead so the url is more manageable? The mapping from hash to query would have to be stored in order to recover/lookup the page after sharing.

It's not uncommon to have things like this currently:

https://global-power-plants.datasettes.com/global-power-plants?sql=select+rowid%2C+country%2C+country_long%2C+name%2C+gppd_idnr%2C+capacity_mw%2C+latitude%2C+longitude%2C+primary_fuel%2C+other_fuel1%2C+other_fuel2%2C+other_fuel3%2C+commissioning_year%2C+owner%2C+source%2C+url%2C+geolocation_source%2C+wepp_id%2C+year_of_capacity_data%2C+generation_gwh_2013%2C+generation_gwh_2014%2C+generation_gwh_2015%2C+generation_gwh_2016%2C+generation_gwh_2017%2C+generation_gwh_2018%2C+generation_gwh_2019%2C+generation_data_source%2C+estimated_generation_gwh_2013%2C+estimated_generation_gwh_2014%2C+estimated_generation_gwh_2015%2C+estimated_generation_gwh_2016%2C+estimated_generation_gwh_2017%2C+estimated_generation_note_2013%2C+estimated_generation_note_2014%2C+estimated_generation_note_2015%2C+estimated_generation_note_2016%2C+estimated_generation_note_2017+from+%5Bglobal-power-plants%5D+order+by+rowid+limit+101

I'm thinking a plugin like https://datasette.io/plugins/datasette-query-files, but could be created and managed from the UI (with the right permissions).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Share button for copying current URL 814595021  
1692210044 https://github.com/simonw/datasette/issues/2143#issuecomment-1692210044 https://api.github.com/repos/simonw/datasette/issues/2143 IC_kwDOBm6k_c5k3RN8 simonw 9599 2023-08-24T18:28:27Z 2023-08-24T18:28:27Z OWNER

Just spotted this: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L328-L332

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L359-L360

Looks to me like that second bit of code doesn't yet handle datasette.yml

This code does though:

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L333-L335

parse_metadata() is clearly a bad name for this function:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/init.py#L980-L990

That @documented decorator indicates that it's part of the documented API used by plugin authors: https://docs.datasette.io/en/1.0a4/internals.html#parse-metadata-content

So we should rename it to something better like parse_json_or_yaml() but keep parse_metadata as an undocumented alias for that to avoid any unnecessary plugin breaks.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
De-tangling Metadata before Datasette 1.0 1855885427  
1692206200 https://github.com/simonw/datasette/issues/2156#issuecomment-1692206200 https://api.github.com/repos/simonw/datasette/issues/2156 IC_kwDOBm6k_c5k3QR4 simonw 9599 2023-08-24T18:25:23Z 2023-08-24T18:25:23Z OWNER

Ran out of time for this, I'll look at the next step next week.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette -s/--setting option for setting nested configuration options 1865649347  
1692201647 https://github.com/simonw/datasette/issues/2156#issuecomment-1692201647 https://api.github.com/repos/simonw/datasette/issues/2156 IC_kwDOBm6k_c5k3PKv simonw 9599 2023-08-24T18:21:53Z 2023-08-24T18:21:53Z OWNER

Oops, that was meant to be a PR. It's just a utility function though so it's safe to land already. I'll do a PR for the actual integration of it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette -s/--setting option for setting nested configuration options 1865649347  
1692186522 https://github.com/simonw/datasette/issues/2156#issuecomment-1692186522 https://api.github.com/repos/simonw/datasette/issues/2156 IC_kwDOBm6k_c5k3Lea simonw 9599 2023-08-24T18:10:04Z 2023-08-24T18:10:04Z OWNER

I have an implementation in https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514 too - I'm going to land that as a PR.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette -s/--setting option for setting nested configuration options 1865649347  
1692182910 https://github.com/simonw/datasette/issues/2143#issuecomment-1692182910 https://api.github.com/repos/simonw/datasette/issues/2143 IC_kwDOBm6k_c5k3Kl- simonw 9599 2023-08-24T18:06:57Z 2023-08-24T18:08:17Z OWNER

The other thing that could work is something like this: bash export AUTH_TOKENS_DB="tokens" datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB So -e is an alternative version of -s which reads from the named environment variable instead of having the value provided directly as the second value in the pair.

I quite like this, because it could replace the really ugly $ENV pattern we have in plugin configuration at the moment: https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values yaml plugins: datasette-auth-github: client_secret: $env: GITHUB_CLIENT_SECRET

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
De-tangling Metadata before Datasette 1.0 1855885427  
1692180683 https://github.com/simonw/datasette/issues/2143#issuecomment-1692180683 https://api.github.com/repos/simonw/datasette/issues/2143 IC_kwDOBm6k_c5k3KDL simonw 9599 2023-08-24T18:05:17Z 2023-08-24T18:05:17Z OWNER

That's a really good call, thanks @rclement - environment variable configuration totally makes sense here.

Need to figure out the right syntax for that. Something like this perhaps:

bash DATASETTE_CONFIG_PLUGINS='{"datasette-ripgrep": ...}' Hard to know how to make this nestable though. I considered this: bash DATASETTE_CONFIG_PLUGINS_DATASETTE_RIPGREP_PATH='/path/to/code/' But that doesn't work, because how does the processing code know that it should split on _ for most of the tokens but NOT split DATASETTE_RIPGREP, instead treating that as datasette-ripgrep?

I checked and - is not a valid character in an environment variable, at least in zsh on macOS: % export FOO_BAR-BAZ=1 export: not valid in this context: FOO_BAR-BAZ

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
De-tangling Metadata before Datasette 1.0 1855885427  
1691845306 https://github.com/simonw/datasette/pull/2154#issuecomment-1691845306 https://api.github.com/repos/simonw/datasette/issues/2154 IC_kwDOBm6k_c5k14K6 simonw 9599 2023-08-24T14:57:39Z 2023-08-24T14:57:39Z OWNER

Notes on manual testing so far - it looks like this might be working! - https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Cascade for restricted token view-table/view-database/view-instance operations 1865281760  
1691842259 https://github.com/simonw/datasette/issues/2102#issuecomment-1691842259 https://api.github.com/repos/simonw/datasette/issues/2102 IC_kwDOBm6k_c5k13bT simonw 9599 2023-08-24T14:55:54Z 2023-08-24T14:55:54Z OWNER

So what's needed to finish this is: - Tests that demonstrate that nothing is revealed that shouldn't be by tokens restricted in this way - Similar tests for other permissions like create-table that check that they work (and don't also need view-instance etc). - Documentation

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
API tokens with view-table but not view-database/view-instance cannot access the table 1805076818  
1691824713 https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713 https://api.github.com/repos/simonw/datasette/issues/2102 IC_kwDOBm6k_c5k1zJJ simonw 9599 2023-08-24T14:45:49Z 2023-08-24T14:45:49Z OWNER

I tested this out against a Datasette Cloud instance. I created a restricted token and tested it like this: bash curl -H "Authorization: Bearer $TOKEN" \ 'https://$INSTANCE/-/actor.json' | jq json { "actor": { "id": "245", "token": "dsatok", "token_id": 2, "_r": { "r": { "data": { "all_stocks": [ "vt" ] } } } } } It can access the all_stocks demo table: bash curl -H "Authorization: Bearer $TOKEN" \ 'https://$INSTANCE/data/all_stocks.json?_size=1' | jq json { "ok": true, "next": "1", "rows": [ { "rowid": 1, "Date": "2013-01-02", "Open": 79.12, "High": 79.29, "Low": 77.38, "Close": 78.43, "Volume": 140124866, "Name": "AAPL" } ], "truncated": false } Accessing the database returns just information about that table, even though other tables exist: bash curl -H "Authorization: Bearer $TOKEN" \ 'https://$INSTANCE/data.json?_size=1' json { "database": "data", "private": true, "path": "/data", "size": 3796992, "tables": [ { "name": "all_stocks", "columns": [ "Date", "Open", "High", "Low", "Close", "Volume", "Name" ], "primary_keys": [], "count": 8813, "hidden": false, "fts_table": null, "foreign_keys": { "incoming": [], "outgoing": [] }, "private": true } ], "hidden_count": 0, "views": [], "queries": [], "allow_execute_sql": false, "table_columns": {} } And hitting the top-level /.json thing does the same - it reveals that table but not any of the other tables or databases: bash curl -H "Authorization: Bearer $TOKEN" \ 'https://$INSTANCE/.json?_size=1' json { "data": { "name": "data", "hash": null, "color": "8d777f", "path": "/data", "tables_and_views_truncated": [ { "name": "all_stocks", "columns": [ "Date", "Open", "High", "Low", "Close", "Volume", "Name" ], "primary_keys": [], "count": null, "hidden": false, "fts_table": null, "num_relationships_for_sorting": 0, "private": false } ], "tables_and_views_more": false, "tables_count": 1, "table_rows_sum": 0, "show_table_row_counts": false, "hidden_table_rows_sum": 0, "hidden_tables_count": 0, "views_count": 0, "private": false } }

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
API tokens with view-table but not view-database/view-instance cannot access the table 1805076818  
1691788400 https://github.com/simonw/datasette/pull/2154#issuecomment-1691788400 https://api.github.com/repos/simonw/datasette/issues/2154 IC_kwDOBm6k_c5k1qRw simonw 9599 2023-08-24T14:25:56Z 2023-08-24T14:25:56Z OWNER

Can be tested with: bash pip install https://github.com/simonw/datasette/archive/6d57a8c23043e99b27f7a2afbe58f4d58815fd51.zip

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Cascade for restricted token view-table/view-database/view-instance operations 1865281760  
1691779180 https://github.com/simonw/datasette/issues/2153#issuecomment-1691779180 https://api.github.com/repos/simonw/datasette/issues/2153 IC_kwDOBm6k_c5k1oBs simonw 9599 2023-08-24T14:21:03Z 2023-08-24T14:21:03Z OWNER

datasette serve currently only has a --get - for this to be really useful it needs to grow --post and maybe other verbs too.

Which is a good argument for moving this functionality to datasette client get ... instead.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette --get --actor option 1865232341  
1691767797 https://github.com/simonw/datasette/pull/2152#issuecomment-1691767797 https://api.github.com/repos/simonw/datasette/issues/2152 IC_kwDOBm6k_c5k1lP1 simonw 9599 2023-08-24T14:15:10Z 2023-08-24T14:15:10Z OWNER

This is broken because Sphinx no longer supports Python 3.8.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump the python-packages group with 3 updates 1865174661  
1691763427 https://github.com/simonw/datasette/issues/2153#issuecomment-1691763427 https://api.github.com/repos/simonw/datasette/issues/2153 IC_kwDOBm6k_c5k1kLj simonw 9599 2023-08-24T14:12:43Z 2023-08-24T14:12:43Z OWNER

Annoying that datasette client ... makes a great name both for a plugin that executes simulated queries against a local database (thanks to its similarity to the existing datasette.client Python API) but is also the ideal name for a command for running commands as a client of an external Datasette instance!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette --get --actor option 1865232341  
1691761685 https://github.com/simonw/datasette/issues/2153#issuecomment-1691761685 https://api.github.com/repos/simonw/datasette/issues/2153 IC_kwDOBm6k_c5k1jwV simonw 9599 2023-08-24T14:11:41Z 2023-08-24T14:11:41Z OWNER

Another option: implement this as a plugin, providing a new command like datasette get ...

Or implement datasette client get ... as core commands or a plugin - except that clashes with the datasette client command that https://github.com/simonw/dclient adds (which is a tool for hitting remote Datasette instances, not running simulated queries through a local one).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette --get --actor option 1865232341  
1691758168 https://github.com/simonw/datasette/issues/2102#issuecomment-1691758168 https://api.github.com/repos/simonw/datasette/issues/2102 IC_kwDOBm6k_c5k1i5Y simonw 9599 2023-08-24T14:09:45Z 2023-08-24T14:09:45Z OWNER

I'm going to implement this in a branch to make it easier to test out.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
API tokens with view-table but not view-database/view-instance cannot access the table 1805076818  
1691753489 https://github.com/simonw/datasette/issues/2153#issuecomment-1691753489 https://api.github.com/repos/simonw/datasette/issues/2153 IC_kwDOBm6k_c5k1hwR simonw 9599 2023-08-24T14:07:25Z 2023-08-24T14:09:16Z OWNER

Building that "_r" array is the main reason this would be useful, but it's also fiddly to get right.

datasette create-token has a design for that already: https://docs.datasette.io/en/1.0a4/authentication.html#datasette-create-token datasette create-token root \ --secret mysecret \ --all view-instance \ --all view-table \ --database docs view-query \ --resource docs documents insert-row \ --resource docs documents update-row Adding imitations of those options (excluding --secret, not needed here) to datasette serve would add a LOT of extra options, but it would also make it really convenient to attempt a request with a specific set of restrictions. Not sure if that would be worth the extra --help output or not.

I feel like the names would have to have a common prefix though. Maybe something like this:

bash datasette serve data.db --get `/data/mytable.json' \ --actor-id root \ --r-all view-instance \ --r-database data view-query \ --r-resource data documents update-row Other options could be the longer --restrict-all/--restrict-database/--restrict-resource.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette --get --actor option 1865232341  
1691094870 https://github.com/simonw/datasette/issues/2143#issuecomment-1691094870 https://api.github.com/repos/simonw/datasette/issues/2143 IC_kwDOBm6k_c5kzA9W rclement 1238873 2023-08-24T06:43:40Z 2023-08-24T06:43:40Z NONE

If I may, the "path-like" configuration is great but one thing that would be even greater: allowing the same configuration to be provided using environment variables.

For instance:

datasette -s plugins.datasette-complex-plugin.configs '{"foo": [1,2,3], "bar": "baz"}'

could also be provided using:

export DS_PLUGINS_DATASETTE-COMPLEX-PLUGIN_CONFIGS='{"foo": [1,2,3], "bar": "baz"}' datasette

(I do not like mixing - and _ in env vars but I do not have a best easily reversible example at the moment)

FYI, you could take some inspiration from another great open source data project, Metabase: https://www.metabase.com/docs/latest/configuring-metabase/config-file https://www.metabase.com/docs/latest/configuring-metabase/environment-variables

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
De-tangling Metadata before Datasette 1.0 1855885427  
1691045051 https://github.com/simonw/datasette/issues/2102#issuecomment-1691045051 https://api.github.com/repos/simonw/datasette/issues/2102 IC_kwDOBm6k_c5ky0y7 simonw 9599 2023-08-24T05:51:59Z 2023-08-24T05:51:59Z OWNER

With that fix in place, this works: bash datasette fixtures.db --get '/fixtures/facetable.json' --actor '{ "_r": { "r": { "fixtures": { "facetable": [ "vt" ] } } }, "a": "user" }' But this fails, because it's for a table not explicitly listed: bash datasette fixtures.db --get '/fixtures/searchable.json' --actor '{ "_r": { "r": { "fixtures": { "facetable": [ "vt" ] } } }, "a": "user" }'

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
API tokens with view-table but not view-database/view-instance cannot access the table 1805076818  
1691044283 https://github.com/simonw/datasette/issues/2102#issuecomment-1691044283 https://api.github.com/repos/simonw/datasette/issues/2102 IC_kwDOBm6k_c5ky0m7 simonw 9599 2023-08-24T05:51:02Z 2023-08-24T05:51:02Z OWNER

Also need to confirm that permissions like insert-row, delete-row, create-table etc don't also need special cases to ensure they get through the view-instance etc checks, if those exist for those actions.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
API tokens with view-table but not view-database/view-instance cannot access the table 1805076818  
1691043475 https://github.com/simonw/datasette/issues/2102#issuecomment-1691043475 https://api.github.com/repos/simonw/datasette/issues/2102 IC_kwDOBm6k_c5ky0aT simonw 9599 2023-08-24T05:50:04Z 2023-08-24T05:50:04Z OWNER

On first test this seems to work!

```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 63a66c3c..9303dac8 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -187,6 +187,30 @@ def permission_allowed_actor_restrictions(datasette, actor, action, resource): return None _r = actor.get("_r")

  • Special case for view-instance: it's allowed if there are any view-database

  • or view-table permissions defined

  • if action == "view-instance":
  • database_rules = _r.get("d") or {}
  • for rules in database_rules.values():
  • if "vd" in rules or "view-database" in rules:
  • return None
  • Now check resources

  • resource_rules = _r.get("r") or {}
  • for _database, resources in resource_rules.items():
  • for rules in resources.values():
  • if "vt" in rules or "view-table" in rules:
  • return None +
  • Special case for view-database: it's allowed if there are any view-table permissions

  • defined within that database

  • if action == "view-database":
  • database_name = resource
  • resource_rules = _r.get("r") or {}
  • resources_in_database = resource_rules.get(database_name) or {}
  • for rules in resources_in_database.values():
  • if "vt" in rules or "view-table" in rules:
  • return None + # Does this action have an abbreviation? to_check = {action} permission = datasette.permissions.get(action) ``` Needs a LOT of testing to make sure what it's doing is sensible though.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
API tokens with view-table but not view-database/view-instance cannot access the table 1805076818  
1691037971 https://github.com/simonw/datasette/issues/2102#issuecomment-1691037971 https://api.github.com/repos/simonw/datasette/issues/2102 IC_kwDOBm6k_c5kyzET simonw 9599 2023-08-24T05:42:47Z 2023-08-24T05:42:47Z OWNER

I applied a fun trick to help test this out: diff diff --git a/datasette/cli.py b/datasette/cli.py index 58f89c1c..830f47ef 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -445,6 +445,10 @@ def uninstall(packages, yes): "--token", help="API token to send with --get requests", ) +@click.option( + "--actor", + help="Actor to use for --get requests", +) @click.option("--version-note", help="Additional note to show on /-/versions") @click.option("--help-settings", is_flag=True, help="Show available settings") @click.option("--pdb", is_flag=True, help="Launch debugger on any errors") @@ -499,6 +503,7 @@ def serve( root, get, token, + actor, version_note, help_settings, pdb, @@ -611,7 +616,10 @@ def serve( headers = {} if token: headers["Authorization"] = "Bearer {}".format(token) - response = client.get(get, headers=headers) + cookies = {} + if actor: + cookies["ds_actor"] = client.actor_cookie(json.loads(actor)) + response = client.get(get, headers=headers, cookies=cookies) click.echo(response.text) exit_code = 0 if response.status == 200 else 1 sys.exit(exit_code) This adds a --actor option to datasette ... --get /path which makes it easy to test an API endpoint using a fake actor with a set of _r restrictions.

With that in place I can try this, with a token that has view-instance and view-database and view-table: bash datasette fixtures.db --get '/fixtures/facetable.json' --actor '{ "_r": { "a": [ "vi" ], "d": { "fixtures": [ "vd" ] }, "r": { "fixtures": { "facetable": [ "vt" ] } } }, "a": "user" }' Or this, with a token that just has view-table but is missing the view-database and view-instance: bash datasette fixtures.db --get '/fixtures/facetable.json' --actor '{ "_r": { "r": { "fixtures": { "facetable": [ "vt" ] } } }, "a": "user" }'

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
API tokens with view-table but not view-database/view-instance cannot access the table 1805076818  
1691036559 https://github.com/simonw/datasette/issues/2102#issuecomment-1691036559 https://api.github.com/repos/simonw/datasette/issues/2102 IC_kwDOBm6k_c5kyyuP simonw 9599 2023-08-24T05:40:53Z 2023-08-24T05:40:53Z OWNER

There might be an easier way to solve this. Here's some permission checks that run when hitting /fixtures/facetable.json:

``` permission_allowed: action=view-table, resource=('fixtures', 'facetable'), actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

File "/datasette/views/table.py", line 727, in table_view_traced view_data = await table_view_data(

File "/datasette/views/table.py", line 875, in table_view_data visible, private = await datasette.check_visibility(

File "/datasette/app.py", line 890, in check_visibility await self.ensure_permissions(actor, permissions)

permission_allowed: action=view-database, resource=fixtures, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

File "/datasette/views/table.py", line 727, in table_view_traced view_data = await table_view_data(

File "/datasette/views/table.py", line 875, in table_view_data visible, private = await datasette.check_visibility(

File "/datasette/app.py", line 890, in check_visibility await self.ensure_permissions(actor, permissions)

permission_allowed: action=view-instance, resource=<None>, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

File "/datasette/views/table.py", line 727, in table_view_traced view_data = await table_view_data(

File "/datasette/views/table.py", line 875, in table_view_data visible, private = await datasette.check_visibility(

File "/datasette/app.py", line 890, in check_visibility await self.ensure_permissions(actor, permissions) ``` That's with a token that has the view instance, view database and view table permissions required.

But... what if the restrictions logic said that if you have view-table you automatically also get view-database and view-instance?

Would that actually let people do anything they shouldn't be able to do? I don't think it would even let them see a list of tables that they weren't allowed to visit, so it might be OK.

I'll try that and see how it works.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
API tokens with view-table but not view-database/view-instance cannot access the table 1805076818  
1690955706 https://github.com/simonw/datasette/issues/2147#issuecomment-1690955706 https://api.github.com/repos/simonw/datasette/issues/2147 IC_kwDOBm6k_c5kye-6 jackowayed 18899 2023-08-24T03:54:35Z 2023-08-24T03:54:35Z NONE

That's fair. The best idea I can think of is that if a plugin wanted to limit intensive queries, it could add LIMITs or something. A hook that gives you visibility of queries and maybe the option to reject felt a little more limited than the existing plugin hooks, so I was trying to think of what else one might want to do while looking at to-be-run queries.

But without a real motivating example, I see why you don't want to add that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for database queries that are run 1858228057  
1690800119 https://github.com/simonw/datasette/issues/2143#issuecomment-1690800119 https://api.github.com/repos/simonw/datasette/issues/2143 IC_kwDOBm6k_c5kx4_3 simonw 9599 2023-08-24T00:10:32Z 2023-08-24T00:39:00Z OWNER

Something notable about this design is that, because the values in the key-value pairs are treated as JSON first and then strings only if they don't parse cleanly as JSON, it's possible to represent any structure (including nesting structures) using this syntax. You can do things like this if you need to (settings for an imaginary plugin):

bash datasette data.db \ -s plugins.datasette-complex-plugin.configs '{"foo": [1,2,3], "bar": "baz"}' Which would be equivalent to: yaml plugins: datasette-complex-plugin: configs: foo: - 1 - 2 - 3 bar: baz This is a bit different from a previous attempt I made at the same problem: https://github.com/simonw/json-flatten - that used syntax like foo.bar.[0]$int = 1 to specify an integer as the first item of an array, which is much more complex.

That previous design was meant to support round-trips, so you could take any nested JSON object and turn it into an HTMl form or query string where every value can have its own form field, then turn the result back again.

For the datasette -s key value feature we don't need round-tripping with individual values each editable on their own, so we can go with something much simpler.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
De-tangling Metadata before Datasette 1.0 1855885427  
1690800641 https://github.com/simonw/datasette/issues/2143#issuecomment-1690800641 https://api.github.com/repos/simonw/datasette/issues/2143 IC_kwDOBm6k_c5kx5IB simonw 9599 2023-08-24T00:11:16Z 2023-08-24T00:11:16Z OWNER

@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.

That's a neat example thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
De-tangling Metadata before Datasette 1.0 1855885427  
1690799608 https://github.com/simonw/datasette/issues/2143#issuecomment-1690799608 https://api.github.com/repos/simonw/datasette/issues/2143 IC_kwDOBm6k_c5kx434 pkulchenko 77071 2023-08-24T00:09:47Z 2023-08-24T00:10:41Z NONE

@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
De-tangling Metadata before Datasette 1.0 1855885427  
1690792514 https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514 https://api.github.com/repos/simonw/datasette/issues/2143 IC_kwDOBm6k_c5kx3JC simonw 9599 2023-08-24T00:00:16Z 2023-08-24T00:02:55Z OWNER

I've been thinking about what it might look like to allow command-line arguments to be used to define any of the configuration options in datasette.yml, as alternative and more convenient syntax.

Here's what I've come up with: datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -s plugins.datasette-auth-tokens.manage_tokens_database tokens \ -s plugins.datasette-ripgrep.path "/home/simon/code-to-search" \ -s databases.mydatabase.tables.example_table.sort created \ mydatabase.db tokens.db Which would be equivalent to datasette.yml containing this: yaml plugins: datasette-auth-tokens: manage_tokens: true manage_tokens_database: tokens datasette-ripgrep: path: /home/simon/code-to-search databases: mydatabase: tables: example_table: sort: created settings: sql_time_limit_ms: 1000 Here's a prototype implementation of this: ```python import json from typing import Any, List, Tuple

def _handle_pair(key: str, value: str) -> dict: """ Turn a key-value pair into a nested dictionary. foo, bar => {'foo': 'bar'} foo.bar, baz => {'foo': {'bar': 'baz'}} foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}} foo.bar, "baz" => {'foo': {'bar': 'baz'}} foo.bar, '{"baz": "qux"}' => {'foo': {'bar': "{'baz': 'qux'}"}} """ try: value = json.loads(value) except json.JSONDecodeError: # If it doesn't parse as JSON, treat it as a string pass

keys = key.split('.')
result = current_dict = {}

for k in keys[:-1]:
    current_dict[k] = {}
    current_dict = current_dict[k]

current_dict[keys[-1]] = value
return result

def _combine(base: dict, update: dict) -> dict: """ Recursively merge two dictionaries. """ for key, value in update.items(): if isinstance(value, dict) and key in base and isinstance(base[key], dict): base[key] = _combine(base[key], value) else: base[key] = value return base

def handle_pairs(pairs: List[Tuple[str, Any]]) -> dict: """ Parse a list of key-value pairs into a nested dictionary. """ result = {} for key, value in pairs: parsed_pair = _handle_pair(key, value) result = _combine(result, parsed_pair) return result Exercised like this:python print(json.dumps(handle_pairs([ ("settings.sql_time_limit_ms", "1000"), ("plugins.datasette-auth-tokens.manage_tokens", "true"), ("plugins.datasette-auth-tokens.manage_tokens_database", "tokens"), ("plugins.datasette-ripgrep.path", "/home/simon/code-to-search"), ("databases.mydatabase.tables.example_table.sort", "created"), ]), indent=4)) Output:json { "settings": { "sql_time_limit_ms": 1000 }, "plugins": { "datasette-auth-tokens": { "manage_tokens": true, "manage_tokens_database": "tokens" }, "datasette-ripgrep": { "path": "/home/simon/code-to-search" } }, "databases": { "mydatabase": { "tables": { "example_table": { "sort": "created" } } } } } `` Note that-sisn't currently an option fordatasette serve`.

--setting key value IS an existing option, but it isn't completely compatible with this because it maps directly just to settings.

Although... we could keep compatibility by saying that if you call --setting known_setting value and that known_setting is in this list then we treat it as if you said -s settings.known_setting value instead:

https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/app.py#L114-L204

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
De-tangling Metadata before Datasette 1.0 1855885427  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Queries took 1002.606ms · About: github-to-sqlite
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows