This also makes it simple to separate out secrets.

datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml

Having multiple configs that combine in that way is a really interesting direction.

To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate "settings" from "configuration" is, I believe, an uphill fight.

I'm very keen on separating out the "metadata" - where metadata is the slimmest possible set of things, effectively the data license and the source and the column and table descriptions - from everything else, mainly because I want metadata to be able to travel with the data.

One idea that's been discussed before is having an optional mechanism for storing metadata in the SQLite database file itself - potentially in a _datasette_metadata table. That way you could distribute a DB file and anyone who opened it in Datasette would also see the correct metadata about it.

That's why I'm so keen on splitting out metadata from all of the other stuff - settings and plugin configuration and authentication rules.

So really it becomes "true metadata" v.s. "all of the other junk that's accumulated in metadata and settings.json".

Codecov Report

Patch and project coverage have no change.

Comparison is base (bdf59eb) 92.73% compared to head (b895cd2) 92.73%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Idea: datasette-permissions-debug plugin which simply prints out a stacktrace for every permission check so you can see where in the code they are.

Built this new test: python @pytest.mark.asyncio async def test_view_table_token_can_access_table(perms_ds): actor = { "id": "restricted-token", "token": "dstok", # Restricted to just view-table on perms_ds_two/t1 "_r": {"r": {"perms_ds_two": {"t1": ["vt"]}}}, } cookies = {"ds_actor": perms_ds.client.actor_cookie(actor)} response = await perms_ds.client.get("/perms_ds_two/t1.json", cookies=cookies) assert response.status_code == 200 The test fails. Running it with pytest --pdb let me do this: (Pdb) from pprint import pprint (Pdb) pprint(perms_ds._permission_checks) deque([{'action': 'view-table', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': ('perms_ds_two', 't1'), 'result': None, 'used_default': True, 'when': '2023-08-23T21:59:45.117155'}, {'action': 'view-database', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': 'perms_ds_two', 'result': False, 'used_default': False, 'when': '2023-08-23T21:59:45.117189'}, {'action': 'view-instance', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': None, 'result': False, 'used_default': False, 'when': '2023-08-23T21:59:45.126751'}, {'action': 'debug-menu', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': None, 'result': False, 'used_default': False, 'when': '2023-08-23T21:59:45.126777'}], maxlen=200)

This is the hook in question: https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/hookspecs.py#L108-L110

• True means they are allowed to access it. You only need a singleTrue from a plugin to allow it.
• False means they are not, and just one False from a plugin will deny it (even if another one returned True I think)
• None means that the plugin has no opinion on this question.

I added outline: 3px dotted pink to that form label block to help spot where else it's being used.

Oh interesting, looks like it's over-ridden here too:

Looks like that affects a few forms:

The search form on the table page over-rides it already:

That should be scoped just to the query filters form on the table page.

I'll fix it in main but I'm still going to ship a fix for those plugins separately so they work well before people upgrade to the next Datasette release with this change in it.

https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/static/app.css#L485-L489

Dupe of: - #2097

Codecov Report

Patch coverage has no change and project coverage change: +0.01% :tada:

Comparison is base (17ec309) 92.71% compared to head (a651d3c) 92.73%. Report is 2 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

I'm happy to debug and land a patch if it's welcome.

Yes please! What an odd bug.

I'm less convinced by the "rewrite the query in some way" optional idea. What kind of use-cases can you imagine for that?

My hunch is that it's much more likely to cause weird breakages than it is to allow for useful plugin extensions, but I'm willing to be convinced otherwise.

Interestingly enough there's actually a mechanism that looks like that a bit already: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/views/database.py#L496-L508

That validate_sql_select() function is defined here: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/init.py#L256-L265

Against these constants:

https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/init.py#L223-L253

Which isn't a million miles away from your suggestion to have a hook that can say if the query should be executed or not.

Looks like this PR has been edited by someone other than Dependabot. That means Dependabot can't rebase it - sorry!

If you're happy for Dependabot to recreate it from scratch, overwriting any edits, you can request @dependabot recreate.

@dependabot rebase

Simplest possible solution is to only run the pip install .[docs] bit under Python 3.9+, ditto for the docs tests. I think I'll try that.

Here's why the tests are failing:

https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/.github/workflows/test.yml#L30-L46

It looks like those tests don't actually need Sphinx installed - they install pip install -e '.[test,docs]' to get the other docs dependencies:

https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/setup.py#L70-L80

Sphinx dropped support for Python 3.8 in their version 7.2.0:

• https://www.sphinx-doc.org/en/master/changes.html#release-7-2-0-released-aug-17-2023
• https://www.sphinx-doc.org/en/master/internals/release-process.html#python-version-support-policy
• https://github.com/sphinx-doc/sphinx/pull/11511

This could play havoc with unmerged PRs. I should merge any big ones (like the JavaScript plugins work) first.

Thinking about this again today. Posted about it on Discord: https://discord.com/channels/823971286308356157/823971286941302908/1143729300349132933

I won't enforce it in a pytest test, I'll enforce it with a CI lint check that's also in the Justfile here instead:

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/Justfile#L19-L23

Browse this commit to see the result: https://github.com/simonw/datasette/tree/59a5d336bd4336bc53103922ada4bf726f4336c9

I just tried this again today by dropping this into .isort.cfg: ini [settings] multi_line_output=3 include_trailing_comma=True force_grid_wrap=0 use_parentheses=True line_length=88 known_first_party=datasette And running this in the root of the project: isort . It produced a huge diff, but when I ran the tests: bash pytest -n auto The tests all passed. I'll push a PR.

This branch is WAY out of date.

It's going to be called datasette.json and the concept of metadata will be split out separately. See:

• 2149

This is obsoleted by the work happening here: - #2093

@dependabot recreate

This is a really great start - tests pass, implementation looks clean, the new stubbed documentation page makes sense. Let's land it in main and iterate on it further in future PRs.