issue_comments

This looks relevant:

https://pluggy.readthedocs.io/en/stable/#wrappers

A hookimpl can be marked with the "wrapper" option, which indicates that the function will be called to wrap (or surround) all other normal hookimpl calls. A hook wrapper can thus execute some code ahead and after the execution of all corresponding non-wrappper hookimpls.

This could be the perfect mechanism for implementing this hook, although I still need to figure out how it interacts with streaming.

In working on this I learned that rowid values in SQLite are way less stable than I had thought - in particular, they are often entirely rewritten on a VACUUM:

https://www.sqlite.org/lang_vacuum.html#how_vacuum_works

The VACUUM command may change the ROWIDs of entries in any tables that do not have an explicit INTEGER PRIMARY KEY.

So this fix wasn't as valuable as I thought. I need to move away from ever assuming that a rowid is a useful foreign key for anything.

Used by datasette-short-links here: https://github.com/datasette/datasette-short-links/blob/468c3e25dbe06a8dcba8edda59bc16a18e126a51/datasette_short_links/init.py#L108-L115

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (5d123f0) 95.77% compared to head (b86374f) 95.77%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Landed:

• https://docs.datasette.io/en/latest/plugin_hooks.html#actors-from-ids-datasette-actor-ids
• https://docs.datasette.io/en/latest/internals.html#await-actors-from-ids-actor-ids

I've implemented this hook once now in: - https://github.com/datasette/datasette-remote-actors

And built and tested a debug tool for it in: - https://github.com/datasette/datasette-debug-actors-from-ids

I'm now confident in the design of this plugin hook, I'm going to land it.

I've implemented this hook once now in: - https://github.com/datasette/datasette-remote-actors

And built and tested a debug tool for it in: - https://github.com/datasette/datasette-debug-actors-from-ids

I'm now confident in the design of this plugin hook, I'm going to land it.

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (c263704) 92.68% compared to head (e1c5a9d) 92.69%.

:umbrella: View full report in Codecov by Sentry.

:loudspeaker: Have feedback on the report? Share it here.

I shipped the first version of

• https://github.com/datasette/datasette-remote-actors/issues/1

When I land this plugin in a Datasette release I need to update that repo to depend on the new alpha.

Documentation preview: - https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids - https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids

Documentation preview: - https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids - https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids

Makes sense, thanks for explaining!

I think this is both a plugin hook and a await datasette.actors_from_ids(actor_ids) internal API function that calls it.

Yes! That recreated the bug: ```

  assert previous_rows == next_rows

E AssertionError: assert equals failed E [ [
E (1, '1', 'Paris'), (1, '1', 'Paris'),
E (3, '3', 'New York'), (2, '3', 'New York'),
E (4, '4', 'London'), (3, '4', 'London'),
E ] ... E
```

Oh! Maybe the row ID preservation here is a coincidence because the tables are created from scratch and count 1, 2, 3.

If I delete a row from the table and then insert some more - breaking the rowid sequence - it might show the bug.

I just noticed that the table where I encountered this bug wasn't actually a rowid table after all - it had an id column that was a text primary key.

The reason the rowid was important is that's how the FTS mechanism in Datasette relates FTS entries to their rows.

But I tried this test and it passed, too: python def test_transform_preserves_rowids(fresh_db): fresh_db["places"].insert_all( [ {"id": "1", "name": "Paris", "country": "France"}, {"id": "2", "name": "London", "country": "UK"}, {"id": "3", "name": "New York", "country": "USA"}, ], pk="id", ) previous_rows = list( tuple(row) for row in fresh_db.execute("select rowid, id, name from places") ) # Transform it fresh_db["places"].transform(column_order=("country", "name")) # Should be the same next_rows = list( tuple(row) for row in fresh_db.execute("select rowid, id, name from places") ) assert previous_rows == next_rows

I tried bumping that up to 10,000 rows instead of just 3 but the test still passed.

That's odd, I wrote a test for this just now and it passes already: python def test_transform_preserves_rowids(fresh_db): # Create a rowid table fresh_db["places"].insert_all( ( {"name": "Paris", "country": "France"}, {"name": "London", "country": "UK"}, {"name": "New York", "country": "USA"}, ), ) assert fresh_db["places"].use_rowid previous_rows = list( tuple(row) for row in fresh_db.execute("select rowid, name from places") ) # Transform it fresh_db["places"].transform(column_order=("country", "name")) # Should be the same next_rows = list( tuple(row) for row in fresh_db.execute("select rowid, name from places") ) assert previous_rows == next_rows So maybe I'm wrong about the cause of that bug?

Codecov Report

Patch and project coverage have no change.

Comparison is base (05707aa) 92.68% compared to head (d75b519) 92.68%. Report is 7 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.

:loudspeaker: Have feedback on the report? Share it here.

We can use this here: https://click.palletsprojects.com/en/8.1.x/options/#values-from-environment-variables python @click.option(..., envvar="DATASETTE_INTERNAL")

To test that locally, use this YAML instead: yaml databases: content: allow: id: root tables: releases: allow: true And: ```yaml allow: id: root databases: content: tables: releases: allow: true

This fix didn't work on Datasette Cloud. I used /-/permissions to debug it and saw this:

Only checking view-table is not enough: for my instance on Datasette Cloud the view permission check that should have failed was for the database or instance.

I ran this:

datasette content.db -p 8043 -m fk-auth.yml --root

Against this YAML: yaml databases: content: tables: users: allow: id: root And it worked as it should - here's a screenshot of an anonymous user and a root user viewing the same page:

Should I put the permission check in that undocumented datasette.expand_foreign_keys() method? I think so - it should accept request.actor as one of its arguments.

Relevant code: https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/views/table.py#L1132-L1149

Which calls this undocumented method:

https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/app.py#L938-L973

@maxhawkins curious why you didn't use the stdlib mailbox to parse the mbox files?

Mailbox parses the entire mbox into memory. Using the lower level library lets us stream the emails in one at a time to support larger archives. Both libraries are in the stdlib.

I ran into the same issue on Windows using

datasette publish cloudrun mydatabase.db --service=my-database do do a google cloud publish.

@Rik-de-Kort your fix worked perfectly! Thanks!

I can always go back and delete the temp directories myself :)

@maxhawkins curious why you didn't use the stdlib mailbox to parse the mbox files?

Here's the thinking: setting theDATASETTE_INTERNAL environment variable for your entire machine will cause any time you run datasette x.db to result in a process that shares the same internal.db database as other instances.

You might run more than one instance at once (I often have 4 or 5 going). This would currently break, because they would over-write each other's catalog tables:

https://github.com/simonw/datasette/blob/e4abae3fd7a828625d00c35c316852ffbaa5ef2f/datasette/utils/internal_db.py#L5-L62

The breaking wouldn't be obvious because the catalog tables aren't used by any features yet, but it's still bad.

This convinced us that actually we should move those catalog_ tables OUT of internal.db. The _internal database will be reserved for plugins that want to use it for caching, storing progress, etc.

I think we move them to an in-memory _catalog database which is excluded from ds.databases (like _internal is ) but can be accessed using datasette.get_catalog_database() - similar to datasette.get_internal_database().

So each instance of Datasette gets its own truly private _catalog, which is in-memory and so gets cleared at the end of each process.

An interesting thing that came up about a shared _internal database is that it provides opportunities for things like a notes plugin which allows you to attach notes to any row in any table in a database... where those notes become available to multiple Datasette instances that you might launch on the same laptop.

DATASETTE_INTERNAL would be more consistent with DATASETTE_SECRET.

The test failure was while installing numpy, relating to importing distutils - maybe relevant: - https://github.com/pypa/setuptools/issues/3661

25h Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' ERROR: Exception: Traceback (most recent call last): ... File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 697, in get_requires_for_build_wheel return super().get_requires_for_build_wheel(config_settings=cs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel return self._call_hook('get_requires_for_build_wheel', { ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook raise BackendUnavailable(data.get('traceback', '')) pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend obj = import_module(mod_path) ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1381, in _gcd_import File "<frozen importlib._bootstrap>", line 1354, in _find_and_load File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1381, in _gcd_import File "<frozen importlib._bootstrap>", line 1354, in _find_and_load File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 929, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 994, in exec_module File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "/tmp/pip-build-env-x9nyg3kd/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module> import distutils.core ModuleNotFoundError: No module named 'distutils'

Codecov Report

Patch and project coverage have no change.

Comparison is base (05707aa) 92.68% compared to head (7c9df6e) 92.68%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (fd083e3) 92.68% compared to head (a481ebd) 92.68%. Report is 1 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Surprising error: blacken-docs -l 60 docs/*.rst shell: /usr/bin/bash -e {0} env: pythonLocation: /opt/hostedtoolcache/Python/3.11.4/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib docs/writing_plugins.rst:365: code block parse error Cannot parse: 8:0: <line number missing in source>

Running tests to see if Pyodide works - that was the reason I switched to my click-default-group-wheel package originally.

Hey @simonw I hit the same issue as mentionned by @chmaynard on a fresh install, "/mnt/fixtures.db" doesn't seem to exist in the docker image

Maybe just populate db.memory: bool and db.memory_name: Optional[str] for this, then document them.

Here's a prototype: https://github.com/simonw/sqlite-utils/commit/62f673835c4a66f87cf6f949eaff43c8b014619b

Still needs tests and documentation (and some more thought to make sure it's doing the right thing).

Normally in Python/sqlite3 you de-register a function by passing None to it.

You can't do that with db.register_function() at the moment because a fn of None does something else:

https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L461-L464

For that particular case I realized I'd quite like to have a mechanism for applying functions for a block of code and then de-registering them at the end - a context manager.

I played with this idea a bit:

python with db.register_functions(md5, md5_random): db.query(...)

https://docs.datasette.io/en/latest/writing_plugins.html#plugins-that-define-new-plugin-hooks

Need to make sure the design of this takes streaming responses into account. Those could be pretty tricky here.

I nice thing about asgi_wrapper() is that it handles those already.

The hook could be called register_middleware() and could work like register_routes() and register_commands():

```python @hookspec def register_middleware(datasette): """Register middleware: returns a list of async def middleware functions"""

```

A pattern like this could be interesting: python async def my_middleware(datasette, request, get_response): # Mess with request here if neccessary response = await get_response(request) # mess with response return response The Django pattern is more complicated but does have that mechanism for running one-time configuration prior to defining the middleware() function, which is neat.

This could even be a pair of hooks - process_request() and process_response().

Or could take a leaf from Django, which redesigned middleware to use this pattern instead:

python def simple_middleware(get_response): # One-time configuration and initialization. def middleware(request): # Code to be executed for each request before # the view (and later middleware) are called. response = get_response(request) # Code to be executed for each request/response after # the view is called. return response return middleware Or even borrow an idea from pytest where fixtures can yield in the middle, like this: python @pytest.fixture def sending_user(mail_admin): user = mail_admin.create_user() yield user mail_admin.delete_user(user)

Not sure what to call this. Maybe app_wrapper()?

Or perhaps it's simpler than that, something like this:

python @hookspec def process_response(datasette, request, response): """Last chance to modify the response before it is returned to the client"""

@simonw what do you think about adding a DATASETTE_INTERNAL_DB_PATH env variable, where when defined, is the default location of the internal DB? This means when the --internal flag is NOT provided, Datasette would check to see if DATASETTE_INTERNAL_DB_PATH exists, and if so, uses that as the internal database (and would fallback to an ephemeral memory database)

My rationale: some plugins may require, or strongly encourage, a persistent internal database (datasette-comments, datasette-bookmarks, datasette-link-shortener, etc.). However, for users that have a global installation of Datasette (say from brew install or a global pip install), it would be annoying having to specify --internal every time. So instead, they can just add export DATASETTE_INTERNAL_DB_PATH="/path/to/internal.db" to their bashrc/zshrc/whereever to not have to worry about --internal

This is implemented now: - https://docs.datasette.io/en/stable/internals.html#await-ensure-permissions-actor-permissions

This is going to happen in this tool instead: - https://github.com/simonw/dclient

Documentation preview: https://github.com/simonw/datasette/blob/6321c9c055a640ed6ea98e231dc5813dcde1f773/docs/plugins.rst#controlling-which-plugins-are-loaded

Codecov Report

Patch coverage: 26.66% and project coverage change: -0.17% :warning:

Comparison is base (30b28c8) 92.84% compared to head (5eddf34) 92.68%.

:exclamation: Current head 5eddf34 differs from pull request most recent head 6321c9c. Consider uploading reports for the commit 6321c9c to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Here's the reason for that name disparity:

https://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L54-L65

Note how the distinfo.project_name name is used when available. That seems to work for regularly installed plugins but not for plugins loaded via DATASETTE_LOAD_PLUGINS.

And that's looking things up in plugin_to_distinfo which is populated here:

https://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L37

The load_setuptools_entrypoints() function in Pluggy does this:

python for ep in dist.entry_points: if ( ep.group != group or (name is not None and ep.name != name) # already registered or self.get_plugin(ep.name) or self.is_blocked(ep.name) ): continue plugin = ep.load() self.register(plugin, name=ep.name)

Slight weirdness: I noticed that the output from the datasette plugins command looks like this for plugins loaded with the new environment variable: json { "name": "datasette_pretty_json", "static": false, "templates": false, "version": null, "hooks": [ "render_cell" ] }, That should ideally be datasette-pretty-json, not datasette_pretty_json.

Testing this is going to be a bit of a pain.

I think I'll add a whole separate test block to CI which installs a couple of plugins and then exercises this feature using datasette plugins.

I'll use datasette-init and datasette-json-html just because they are small and simple.

Figured this out so far with the help of GPT-4: https://chat.openai.com/share/0e785865-621b-4fb3-ba05-7449e57c8496

Now this works: bash DATASETTE_LOAD_PLUGINS=datasette-write-ui datasette plugins json [ { "name": "datasette_write_ui", "static": true, "templates": true, "version": null, "hooks": [ "extra_template_vars", "register_routes" ] } ] Or multiple plugins: bash DATASETTE_LOAD_PLUGINS=datasette-write-ui,datasette-pretty-json datasette plugins Outputs: json [ { "name": "datasette_pretty_json", "static": false, "templates": false, "version": null, "hooks": [ "render_cell" ] }, { "name": "datasette_write_ui", "static": true, "templates": true, "version": null, "hooks": [ "extra_template_vars", "register_routes" ] } ]

https://github.com/simonw/datasette/blob/50da908213a0fc405ecd7a40090dfea7a2e7395c/datasette/utils/internal_db.py#L8-L62

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.03% :tada:

Comparison is base (2e28258) 92.82% compared to head (3e49fd3) 92.85%. Report is 3 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

In that last commit I also upgraded Permission from a named tuple to a dataclass, and added a implies_can_view=True private (not documented) option to it.

Last step is to refactor the code to use that new property.

The code to refactor is this: https://github.com/simonw/datasette/blob/d64a9896f743f87b673d58859a0ec16685594e79/datasette/default_permissions.py#L181-L280

I'm going to turn that into a more general restrictions_allow_action function.

I'm going to merge this so we can see how it feels.

However, one important notes about those new core_ tables: If a --internal DB is passed in, that means those core_ tables will persist across multiple Datasette instances. This wasn't the case before, since _internal was always an in-memory database created from scratch.

I'm completely happy for the core_* tables (or datasette_* or some other name) to live in the persisted-to-disk internal.db database, even though they're effectively meant to be an in-memory cache.

I don't think it causes any harm, and it could even be quite useful to have them visible on disk - other applications could read the internal.db database while Datasette itself is running, should they have some weird reason to want to do that!

Having those tables stick around in internal.db after Datasette shuts down could be useful for other debugging activities as well.

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.03% :warning:

Comparison is base (2e28258) 92.82% compared to head (73489ca) 92.79%. Report is 1 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Documentation preview: https://datasette--2162.org.readthedocs.build/en/2162/internals.html#datasette-s-internal-database

Code for this might be cleaner with a Restrictions() class that takes a "_r" dictionary to the constructor and can then answer questions like .any_resource_has_permission("view-table") - where it can resolve aliases etc as well.

Reminder that I also need to confirm that insert-row works if you have it at the instance level, the database level or the resource level in _r.

Should this have implies_can_view=True too? Probably: https://github.com/simonw/datasette/blob/d64a9896f743f87b673d58859a0ec16685594e79/datasette/default_permissions.py#L20-L22

Codecov Report

Patch and project coverage have no change.

Comparison is base (2e28258) 92.82% compared to head (c81b024) 92.82%.

:exclamation: Current head c81b024 differs from pull request most recent head 3529c9c. Consider uploading reports for the commit 3529c9c to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Cog failed!

Since this bumps Sphinx I'm manually reviewing the ReadTheDocs preview a bit, looks good to me: https://datasette--2160.org.readthedocs.build/en/2160/

This pull request was built based on a group rule. Closing it will not ignore any of these versions in future pull requests.

Now solving this here: - #2160

https://github.com/simonw/datasette/issues/2148#issuecomment-1689177556

Simplest possible solution is to only run the pip install .[docs] bit under Python 3.9+, ditto for the docs tests. I think I'll try that.

I figured out why this was failing in: - https://github.com/simonw/datasette/pull/2148

It's because Sphinx dropped support for Python 3.8.

I want to test "for this set of restrictions, does a GET/POST to this path return 200 or 403"?

Here's an existing relevant test:

https://github.com/simonw/datasette/blob/2e2825869fc2655b5fcadc743f6f9dec7a49bc65/tests/test_permissions.py#L616-L666

It's not quite right for this new set of tests though, since they need to be exercising actual endpoints (/.json etc) in order to check that this works correctly.

Documentation: - https://docs.datasette.io/en/latest/cli-reference.html#datasette-get

Codecov Report

Patch coverage: 75.00% and project coverage change: +0.05% :tada:

Comparison is base (527cec6) 92.76% compared to head (a5cbf80) 92.81%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Now a PR: - #2161

Superseded by #2160.

Tips on typing **kwargs: https://adamj.eu/tech/2021/05/11/python-type-hints-args-and-kwargs/

Codecov Report

Patch and project coverage have no change.

Comparison is base (527cec6) 92.76% compared to head (2e45686) 92.76%.

:exclamation: Current head 2e45686 differs from pull request most recent head 5dfa305. Consider uploading reports for the commit 5dfa305 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Closing this issue in favour of this one: - #2157

But yes, I'm a big +1 on this whole plan.

We discussed this in-person this morning and these notes reflect what we talked about perfectly.

I've had so many bugs with plugins that I've written myself that have forgotten to special-case the _internal database when looping through datasette.databases.keys() - removing it from there entirely would help a lot.

Just one tiny disagreement: for datasette-comments I think having it store things in _internal could be an option, but in most cases I expect users to chose NOT to do that - because being able to join against those tables for more advanced queries is going to be super useful.

Show me all rows in foia_requests with at least one associated comment in datasette_comments.comments kind of tihng.

Something to think about, but I hate how long the url is when sharing a custom SQL query. Would it be possible to hash the query and state of a page instead so the url is more manageable? The mapping from hash to query would have to be stored in order to recover/lookup the page after sharing.

It's not uncommon to have things like this currently:

https://global-power-plants.datasettes.com/global-power-plants?sql=select+rowid%2C+country%2C+country_long%2C+name%2C+gppd_idnr%2C+capacity_mw%2C+latitude%2C+longitude%2C+primary_fuel%2C+other_fuel1%2C+other_fuel2%2C+other_fuel3%2C+commissioning_year%2C+owner%2C+source%2C+url%2C+geolocation_source%2C+wepp_id%2C+year_of_capacity_data%2C+generation_gwh_2013%2C+generation_gwh_2014%2C+generation_gwh_2015%2C+generation_gwh_2016%2C+generation_gwh_2017%2C+generation_gwh_2018%2C+generation_gwh_2019%2C+generation_data_source%2C+estimated_generation_gwh_2013%2C+estimated_generation_gwh_2014%2C+estimated_generation_gwh_2015%2C+estimated_generation_gwh_2016%2C+estimated_generation_gwh_2017%2C+estimated_generation_note_2013%2C+estimated_generation_note_2014%2C+estimated_generation_note_2015%2C+estimated_generation_note_2016%2C+estimated_generation_note_2017+from+%5Bglobal-power-plants%5D+order+by+rowid+limit+101

I'm thinking a plugin like https://datasette.io/plugins/datasette-query-files, but could be created and managed from the UI (with the right permissions).

Just spotted this: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L328-L332

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L359-L360

Looks to me like that second bit of code doesn't yet handle datasette.yml

This code does though:

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L333-L335

parse_metadata() is clearly a bad name for this function:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/init.py#L980-L990

That @documented decorator indicates that it's part of the documented API used by plugin authors: https://docs.datasette.io/en/1.0a4/internals.html#parse-metadata-content

So we should rename it to something better like parse_json_or_yaml() but keep parse_metadata as an undocumented alias for that to avoid any unnecessary plugin breaks.

Ran out of time for this, I'll look at the next step next week.

Oops, that was meant to be a PR. It's just a utility function though so it's safe to land already. I'll do a PR for the actual integration of it.

I have an implementation in https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514 too - I'm going to land that as a PR.

The other thing that could work is something like this: bash export AUTH_TOKENS_DB="tokens" datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB So -e is an alternative version of -s which reads from the named environment variable instead of having the value provided directly as the second value in the pair.

I quite like this, because it could replace the really ugly $ENV pattern we have in plugin configuration at the moment: https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values yaml plugins: datasette-auth-github: client_secret: $env: GITHUB_CLIENT_SECRET

That's a really good call, thanks @rclement - environment variable configuration totally makes sense here.

Need to figure out the right syntax for that. Something like this perhaps:

bash DATASETTE_CONFIG_PLUGINS='{"datasette-ripgrep": ...}' Hard to know how to make this nestable though. I considered this: bash DATASETTE_CONFIG_PLUGINS_DATASETTE_RIPGREP_PATH='/path/to/code/' But that doesn't work, because how does the processing code know that it should split on _ for most of the tokens but NOT split DATASETTE_RIPGREP, instead treating that as datasette-ripgrep?

I checked and - is not a valid character in an environment variable, at least in zsh on macOS: % export FOO_BAR-BAZ=1 export: not valid in this context: FOO_BAR-BAZ

Notes on manual testing so far - it looks like this might be working! - https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713

So what's needed to finish this is: - Tests that demonstrate that nothing is revealed that shouldn't be by tokens restricted in this way - Similar tests for other permissions like create-table that check that they work (and don't also need view-instance etc). - Documentation

I tested this out against a Datasette Cloud instance. I created a restricted token and tested it like this: bash curl -H "Authorization: Bearer $TOKEN" \ 'https://$INSTANCE/-/actor.json' | jq json { "actor": { "id": "245", "token": "dsatok", "token_id": 2, "_r": { "r": { "data": { "all_stocks": [ "vt" ] } } } } } It can access the all_stocks demo table: bash curl -H "Authorization: Bearer $TOKEN" \ 'https://$INSTANCE/data/all_stocks.json?_size=1' | jq json { "ok": true, "next": "1", "rows": [ { "rowid": 1, "Date": "2013-01-02", "Open": 79.12, "High": 79.29, "Low": 77.38, "Close": 78.43, "Volume": 140124866, "Name": "AAPL" } ], "truncated": false } Accessing the database returns just information about that table, even though other tables exist: bash curl -H "Authorization: Bearer $TOKEN" \ 'https://$INSTANCE/data.json?_size=1' json { "database": "data", "private": true, "path": "/data", "size": 3796992, "tables": [ { "name": "all_stocks", "columns": [ "Date", "Open", "High", "Low", "Close", "Volume", "Name" ], "primary_keys": [], "count": 8813, "hidden": false, "fts_table": null, "foreign_keys": { "incoming": [], "outgoing": [] }, "private": true } ], "hidden_count": 0, "views": [], "queries": [], "allow_execute_sql": false, "table_columns": {} } And hitting the top-level /.json thing does the same - it reveals that table but not any of the other tables or databases: bash curl -H "Authorization: Bearer $TOKEN" \ 'https://$INSTANCE/.json?_size=1' json { "data": { "name": "data", "hash": null, "color": "8d777f", "path": "/data", "tables_and_views_truncated": [ { "name": "all_stocks", "columns": [ "Date", "Open", "High", "Low", "Close", "Volume", "Name" ], "primary_keys": [], "count": null, "hidden": false, "fts_table": null, "num_relationships_for_sorting": 0, "private": false } ], "tables_and_views_more": false, "tables_count": 1, "table_rows_sum": 0, "show_table_row_counts": false, "hidden_table_rows_sum": 0, "hidden_tables_count": 0, "views_count": 0, "private": false } }

Can be tested with: bash pip install https://github.com/simonw/datasette/archive/6d57a8c23043e99b27f7a2afbe58f4d58815fd51.zip

datasette serve currently only has a --get - for this to be really useful it needs to grow --post and maybe other verbs too.

Which is a good argument for moving this functionality to datasette client get ... instead.

This is broken because Sphinx no longer supports Python 3.8.

Annoying that datasette client ... makes a great name both for a plugin that executes simulated queries against a local database (thanks to its similarity to the existing datasette.client Python API) but is also the ideal name for a command for running commands as a client of an external Datasette instance!

Another option: implement this as a plugin, providing a new command like datasette get ...

Or implement datasette client get ... as core commands or a plugin - except that clashes with the datasette client command that https://github.com/simonw/dclient adds (which is a tool for hitting remote Datasette instances, not running simulated queries through a local one).