Exception: Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.

I think we should link directly to documentation that tells people how to perform this upgrade.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/2190#issuecomment-1759947021

Sometimes a plugin may want to add content to an existing default template - for example datasette-search-all adds a new search box at the top of index.html. I also want datasette-upload-csvs to add a CTA on the database.html page: https://github.com/simonw/datasette-upload-csvs/issues/18

Currently plugins can do this by providing a new version of the index.html template - but if multiple plugins try to do that only one of them will succeed.

It would be better if there were known areas of those templates which plugins could add additional content to, such that multiple plugins can use the same spot.

The cause of this bug: - https://github.com/simonw/datasette-export-notebook/issues/17

Is that datasette-export-notebook was consulting data["filtered_table_rows_count"] in the render output plugin function in order to show the total number of rows that would be exported.

That field is no longer available by default - the "count" field is only available if ?_extra=count was passed.

It would be useful if plugins like this could access the total count on demand, should they need to.

This idea first came up here: - https://github.com/simonw/datasette/issues/2112#issuecomment-1652751140

If datasette.render_template(...) takes an optional Context subclass as an alternative to a context dictionary, I could then use dataclasses to define the context made available to specific templates - which then gives me something I can use to help document what they are.

Also refs: - https://github.com/simonw/datasette/issues/1510

I ended up duplicating it here: https://github.com/simonw/datasette/blob/7532feb424b1dce614351e21b2265c04f9669fe2/datasette/views/database.py#L164-L167

I think it should move to datasette.render_template() - and maybe have a renamed template variable too.

https://latest.datasette.io/_memory.csv?sql=select+blah is a blank page right now:

bash curl -I 'https://latest.datasette.io/_memory.csv?sql=select+blah' HTTP/2 200 access-control-allow-origin: * access-control-allow-headers: Authorization, Content-Type access-control-expose-headers: Link access-control-allow-methods: GET, POST, HEAD, OPTIONS access-control-max-age: 3600 content-type: text/plain; charset=utf-8 x-databases: _memory, _internal, fixtures, fixtures2, extra_database, ephemeral date: Mon, 07 Aug 2023 23:12:15 GMT server: Google Frontend

Originally posted by @simonw in https://github.com/simonw/datasette/issues/2118#issuecomment-1668688947

This PR changes the display of facets to hopefully make them more readable.

Before | After ---|--- |

Documented context plus protective unit tests. Goal is that custom templates built for 1.x will not break without a 2.x release.

From a discussion on Discord: https://discord.com/channels/823971286308356157/996877076982415491/1082789517062320138

The key idea is to make it easier for people to implement where id in (...) that's populated from query string arguments.

What if you could add ?id=11&id=32&id=62 to the URL and have that made available as a list that can be used in the query?

Datasette views currently work by creating a set of data that should be returned as JSON, then defining an additional, optional template_data() function which is called if the view is being rendered as HTML.

This template_data() function calculates extra template context variables which are necessary for the HTML view but should not be included in the JSON.

Example of how that is used today: https://github.com/simonw/datasette/blob/2b79f2bdeb1efa86e0756e741292d625f91cb93d/datasette/views/table.py#L672-L704

With features like Facets in #255 I'm beginning to want to move more items into the template_data() - in the case of facets it's the suggested_facets array. This saves that feature from being calculated (involving several SQL queries) for the JSON case where it is unlikely to be used.

But... as an API user, I want to still optionally be able to access that information.

Solution: Add a ?_extra=suggested_facets&_extra=table_metadata argument which can be used to optionally request additional blocks to be added to the JSON API.

Then redefine as many of the current template_data() features as extra arguments instead, and teach Datasette to return certain extras by default when rendering templates.

This could allow the JSON representation to be slimmed down further (removing e.g. the table_definition and view_definition keys) while still making that information available to API users who need it.

e.g. first column on this page - cog menu has no option to facet.

https://datasette.io/content/tools

When looping through databases in the index.html template, I'd like to print the description of each database alongside its name. But it appears that isn't passed in from the view, unless I'm missing it. It would be great to have that.

Datasette currently uses the same table layout on the row pages as it does on the table pages:

https://datasette.io/content/pypi_packages?_sort=name&name__exact=datasette-column-inspect

https://datasette.io/content/pypi_packages/datasette-column-inspect

If you shrink down to mobile width you get this instead, on both of those pages:

I think that view, which I think of as the "card view", is plain better if you're looking at just a single row - and it (or a variant of it) should be the default presentation on the row page.

I just realized that when I ship Datasette 1.0 there may be automated deployments out there which could deploy the 1.0 version by accident, potentially breaking any customizations that aren't compatible with the 1.0 changes.

I can hopefully help avoid that by shipping one last entry in the 0.x series that ensures datasette publish pins to <1.0 when it installs Datasette itself.

I'd like to implement this by first extending the register_output_renderer() hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1096#issuecomment-732542285

I hadn't run into an issue with plugins like datasette-template-sql interfering with the existing context for other features before! Definitely not a good thing.

Originally posted by @simonw in https://github.com/simonw/datasette-write/issues/6#issuecomment-1379490596

Decision: as an initial fix I'm going to de-duplicate those keys by using tags__array etc - with a _2 on the end if that key is already used.

I'll open a separate issue to redesign this better for Datasette 1.0.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/625#issuecomment-996130862

Idea came from a conversation with @hydrosquall - what if a Datasette plugin could say "I'd like the JSON for a page to be included in a variable on the HTML page"?

datasette-cluster-map already needs this - the first thing it does when the page loads is fetch() a JSON representation of that same data.

This idea fits with my overall goals to unify the JSON and HTML context too.

Refs: - #1711

Split from: - #1931 - #1878

This is a bit tricky because the code needs to figure out what the primary keys are for an item, and whether or not rowid should be included.

Idea from here: - https://github.com/simonw/sqlite-utils/issues/516

Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions

This method allows multiple permissions to be checked at onced. It raises a datasette.Forbidden exception if any of the checks are denied before one of them is explicitly granted.

This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns True or not a single one of them returns False:

That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827

While working on this issue: - #1896

I realized that Datasette has error classes scattered around a fair bit, including some in the datasette.utils.asgi module for some reason.

I should clean these up.

This is a strong argument for extracting the Heroku support out to a plugin - it would allow this to be fixed with a plugin release without needing to push a full release of Datasette itself.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1905#issuecomment-1320678715

For datasette-copyable I want to show breadcrumbs that take database/instance permissions into account, so I'm removing {% block nav %} entirely and replacing it with this:

html+jinja {% block crumbs %} {{ crumbs.nav(request=request, database=database, table=table) }} {% endblock %}

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1901#issuecomment-1319588163

I should document this.

This can drive the upgrade of the register_output_renderer hook to be able to handle streaming all rows in a large query.

This is a good idea, and one that needs to happen before Datasette 1.0:

While you are adding features, would you be future-proofing your APIs if you switched over some arguments over to keyword-only arguments or would that be too disruptive?

Thinking out loud:

async def render_template( self, templates, *, context=None, plugin_context=None, request=None, view_name=None ): Originally posted by @jefftriplett in https://github.com/simonw/datasette/issues/1817#issuecomment-1256781274

See https://github.com/simonw/datasette-publish-now/issues/9#issuecomment-618155764 - the --extra-options mechanism is in practice just used to set --config options in data that you publish, but that means you end up with pretty messy looking commands:

datasette publish my.db --extra-options="--config default_page_size:50 --config sql_time_limit_ms:3500"

A neater design would be to support --config as an option for datasette publish directly:

datasette publish my.db --config default_page_size:50 --config sql_time_limit_ms:3500

As part of the preparation for the 1.0 release, test all existing known plugins against the alpha.

For any that break, submit a PR suggesting they pin to a version <1.0 - and include a link to the documentation on how to upgrade the plugin for 1.0.

Original title: Set content type for CSV so that browsers will attempt to download instead opening in the browser

Right now, if the user clicks on the CSV related to a <s>table or a</s> query, the response header for the content type is

"content-type: text/plain; charset=utf-8"

Most browsers will try to open a file with this content-type in the browser.

This is not what most people want to do, and lots of folks don't know that if they want to download the CSV and open it in the a spreadsheet program they next need to save the page through their browser.

It would be great if the response header could be something like

'Content-type: text/csv'); 'Content-disposition: attachment;filename=MyVerySpecial.csv');

which would lead browsers to open a download dialog.

Permission checks right now can express one of three opinions:

• False means "so not grant this permisson"
• True means "grant this permission"
• None means "I have no opinion"

But... there's also a concept of a "default" for a given permission check, which might be False or True.

I worry this is too complicated. Could this be simplified before 1.0? In particular the default concept.

See also: - #1676

Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:

pandas.read_json("https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on", lines=True)

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1101#issuecomment-755128038

https://latest.datasette.io/.json

Currently: json { "_memory": { "name": "_memory", "hash": null, "color": "a6c7b9", "path": "/_memory", "tables_and_views_truncated": [], "tables_and_views_more": false, "tables_count": 0, "table_rows_sum": 0, "show_table_row_counts": false, "hidden_table_rows_sum": 0, "hidden_tables_count": 0, "views_count": 0, "private": false }, "fixtures": { "name": "fixtures", "hash": "645005884646eb941c89997fbd1c0dd6be517cb1b493df9816ae497c0c5afbaa", "color": "645005", "path": "/fixtures", "tables_and_views_truncated": [ { "name": "compound_three_primary_keys", "columns": [ "pk1", "pk2", "pk3", "content" ], "primary_keys": [ "pk1", "pk2", "pk3" ], "count": 1001, "hidden": false, "fts_table": null, "num_relationships_for_sorting": 0, "private": false }, As-of this issue the "path" key is confusing, it doesn't match what https://latest.datasette.io/-/databases returns:

• 1668

I think the way to get rid of most of the remaining complexity in DataView is to refactor how CSV stuff works - pulling it in line with other export factors and extracting the streaming mechanism. Opening a fresh issue for that.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1660#issuecomment-1073355032

Can be part of #870 - refactoring existing views to use register_routes().

I'm going to put the new check_permissions() method on BaseView as well. If I want that method to be available to plugins I can do so by turning that BaseView class into a documented API that plugins are encouraged to use themselves. Originally posted by @simonw in https://github.com/simonw/datasette/issues/832#issuecomment-651995453

I realized that one of the reasons I'm having trouble committing to nailing down the JSON API for 1.0 is that I don't use it much myself - I use the ?_shape=array one quite often, but I don't have any projects that are using the default, more fully-featured API.

As an experiment I built a Web Component for embedding Datasette tables on pages - https://github.com/simonw/datasette-table - and I think it's actually going to be a really useful tool for helping me dog food the v1.0 API design.

Did a search on the issues here and didn't find anything related to what I want.

I want to have information that is on the database level of the JSON like title, source and source_url, and use it on the index page.

I tried some small tweaks on the python and html files, but failed to get that result.

Is there a way? Thanks!

https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/datasette/app.py#L297-L328

The new JSON API in a stable, documented form.

Split from #878. The current TableView class is by far the most complex part of Datasette, and the most difficult to work on: https://github.com/simonw/datasette/blob/0.59.2/datasette/views/table.py

In #878 I started exploring a new pattern for building views. In doing so it became clear that TableView is the first beast that I need to slay - if I can refactor that into something neat the pattern for building other views will emerge as a natural consequence.

I've been trying to build this as a register_routes() plugin, as originally suggested in #870 - though unfortunately it looks like those plugins can't replace existing Datasette default views at the moment, see #1517. [UPDATE: I was wrong about this, plugins can over-ride default views just fine]

I also know that I want to have a fully documented template context for table.html as a major step on the way to Datasette 1.0, see #1510.

All of this adds up to the TableView factor being a major project that will unblock a whole flurry of other things - so I'm going to work on that in this separate issue.

The generated Dockerfile currently looks something like this: ```Dockerfile FROM python:3.8 COPY . /app WORKDIR /app

ENV DATASETTE_SECRET 'edab49cbc5d5f6f33238f54852037e3fee710821960b73edd2ce743454182ae2' RUN pip install -U datasette datasette-auth-passwords datasette-tiddlywiki datasette-graphql RUN datasette inspect fixtures.db other.db --inspect-file inspect-data.json ENV PORT 8080 EXPOSE 8080 CMD datasette serve --host 0.0.0.0 -i fixtures.db -i other.db --cors --inspect-file inspect-data.json --metadata metadata.json --create --port $PORT /data/*.db `` This is still on Python 3.8, and it generates a pretty large image compared to theDockerfile` used for https://hub.docker.com/datasetteproject/datasette - https://github.com/simonw/datasette/blob/0.60.2/Dockerfile

Here's the code that generates it: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/utils/init.py#L389-L400

Datasette's documentation is aimed at people who install and configure it.

What about end users of preconfigured and deployed Datasette instances?

Something that can be linked to from the Datasette UI would be really useful.

What I'm seeing

With default_cache_ttl = 86400, I see the following:

A table view returns Cache-control: max-age=86400:

A static asset returns no Cache-control header:

What I expected to see

I expected the static asset to return a Cache-control header indicating that this response can be cached.

Why this matters

I'm productionising a Datasette deployment right now and was looking into putting it behind a Varnish instance. I was surprised to see requests for static assets being served from Datasette rather than Varnish, this is what led me to look more closely at the response headers.

While Datasette serves those static assets pretty quickly, I don't see why Datasette should serve them. By their nature, static assets like images and JS files are very cacheable, so it should be easy to serve them from a cache like Varnish.

(Note that Varnish can easily be configured to override this header, enabling caching for static assets. But it would be better if this override was not necessary.)

Discussion

It seems clear to me that serving static assets without a Cache-control header is not ideal.

I see two options here:

A. Static assets use the same logic as table / SQL views to set the Cache-control header based on default_cache_ttl. B. An additional setting for static assets is introduced (default_static_cache_ttl, say).

It would be nice if the --cors option accepted an origin regex to more securely allow secure local development.

As an example, Observable notebooks namespace every user's notebooks by their username and user content is served from username.observableusercontent.com, so you would set --cors-origin username.observableusercontent.com to restrict access to a local development Datasette instance to only your own notebooks, rather than exposing the data to any website that makes a request.

Thank you for all of your work on Datasette!

Follows: - #1176

Can use /database/-/... namespace from #296

Nothing uses this plugin hook yet, so the design is not yet proven.

I'm going to build a real plugin against it and use that process to inform any design changes that may need to be made.

I'll add a warning about this to the documentation.

Suggested by @frankieroberto in https://github.com/simonw/datasette/issues/782#issuecomment-782746755

Development Status :: 5 - Production/Stable

I need to perform a detailed review of the plugin interface - especially the plugin hooks like register_facet_classes() which I don't yet have complete confidence in.

If I was prone to over-thinking (which I am) I'd note that allow_facet and allow_download and allow_csv_stream are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent.

So maybe there's a future task where I introduce those as both permissions and metadata "allow_x" blocks, then rename the settings themselves to be called default_allow_facet and default_allow_download and default_allow_csv_stream.

If I was going to do that I should get it in before Datasette 1.0.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1409#issuecomment-890400425

I want to theme https://datasette.io/ so that when you visit https://datasette.io/content (the Datasette UI part of it) the navigation from the parent site is used.

I tried dropping in a base.html template like this:

```html {% extends "page_base.html" %}

{% block base_extra_head %}

<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> {% for url in extra_css_urls %} <link rel="stylesheet" href="{{ url.url }}"{% if url.sri %} integrity="{{ url.sri }}" crossorigin="anonymous"{% endif %}> {% endfor %} {% for url in extra_js_urls %} <script src="{{ url.url }}"{% if url.sri %} integrity="{{ url.sri }}" crossorigin="anonymous"{% endif %}></script> {% endfor %} {% block extra_head %}{% endblock %} {% endblock %}

{% block extra_body_end %} {% include "_close_open_menus.html" %}

{% for body_script in body_scripts %} <script>{{ body_script }}</script> {% endfor %} {% endblock %} ``` But this resulted in pages looking like this:

Note that the cog menu is broken and the filter UI is unstyled. To get these working correctly I would need to copy over a whole lot of Datasette's default CSS - and that means that when Datasette changes in the future those pages could break in subtle ways.

Split out from #460 - in order to support large numbers of connected databases the homepage needs to be paginated.

I added a deprecation warning to this in #992.

In #998 I made a change that affected the table page but didn't affect the query page because I incorrectly assumed they shared rendering logic.

Documentation says that the old dictionary mechanism will be deprecated by 1.0:

https://github.com/simonw/datasette/blob/799ecae94824640bdff21f86997f69844048d5c3/docs/plugin_hooks.rst#L460 Originally posted by @simonw in https://github.com/simonw/datasette/issues/953#issuecomment-682312494

--name works inconsistently across the different publish providers - on Cloud Run you should use --service instead for example. Need to review it across all of them and either remove it or clarify what it does.

At the moment the page returning a custom SQL query shows the JSON and CSV APIs, but not the multiple JSON shapes. However, adding the _shape parameter to the JSON API URL manually still works, so perhaps there should be consistency in the interface by having the same "Advanced Export" box for custom SQL queries.