Exception: Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.

I think we should link directly to documentation that tells people how to perform this upgrade.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/2190#issuecomment-1759947021

Sometimes a plugin may want to add content to an existing default template - for example datasette-search-all adds a new search box at the top of index.html. I also want datasette-upload-csvs to add a CTA on the database.html page: https://github.com/simonw/datasette-upload-csvs/issues/18

Currently plugins can do this by providing a new version of the index.html template - but if multiple plugins try to do that only one of them will succeed.

It would be better if there were known areas of those templates which plugins could add additional content to, such that multiple plugins can use the same spot.

The cause of this bug: - https://github.com/simonw/datasette-export-notebook/issues/17

Is that datasette-export-notebook was consulting data["filtered_table_rows_count"] in the render output plugin function in order to show the total number of rows that would be exported.

That field is no longer available by default - the "count" field is only available if ?_extra=count was passed.

It would be useful if plugins like this could access the total count on demand, should they need to.

It is increasingly being useful configuration options, when it started out as purely metadata.

Could cause confusion with the --config mechanism though - maybe that should be called "settings" instead?

I think this has been a bug for a long time.

https://latest.datasette.io/ currently shows:

Those colors are based on a hash of the database name. But when you click through to https://latest.datasette.io/fixtures

It's red on all sub-pages too.

This idea first came up here: - https://github.com/simonw/datasette/issues/2112#issuecomment-1652751140

If datasette.render_template(...) takes an optional Context subclass as an alternative to a context dictionary, I could then use dataclasses to define the context made available to specific templates - which then gives me something I can use to help document what they are.

Also refs: - https://github.com/simonw/datasette/issues/1510

I ended up duplicating it here: https://github.com/simonw/datasette/blob/7532feb424b1dce614351e21b2265c04f9669fe2/datasette/views/database.py#L164-L167

I think it should move to datasette.render_template() - and maybe have a renamed template variable too.

https://latest.datasette.io/_memory.csv?sql=select+blah is a blank page right now:

bash curl -I 'https://latest.datasette.io/_memory.csv?sql=select+blah' HTTP/2 200 access-control-allow-origin: * access-control-allow-headers: Authorization, Content-Type access-control-expose-headers: Link access-control-allow-methods: GET, POST, HEAD, OPTIONS access-control-max-age: 3600 content-type: text/plain; charset=utf-8 x-databases: _memory, _internal, fixtures, fixtures2, extra_database, ephemeral date: Mon, 07 Aug 2023 23:12:15 GMT server: Google Frontend

Originally posted by @simonw in https://github.com/simonw/datasette/issues/2118#issuecomment-1668688947

Documented context plus protective unit tests. Goal is that custom templates built for 1.x will not break without a 2.x release.

From a discussion on Discord: https://discord.com/channels/823971286308356157/996877076982415491/1082789517062320138

The key idea is to make it easier for people to implement where id in (...) that's populated from query string arguments.

What if you could add ?id=11&id=32&id=62 to the URL and have that made available as a list that can be used in the query?

Datasette views currently work by creating a set of data that should be returned as JSON, then defining an additional, optional template_data() function which is called if the view is being rendered as HTML.

This template_data() function calculates extra template context variables which are necessary for the HTML view but should not be included in the JSON.

Example of how that is used today: https://github.com/simonw/datasette/blob/2b79f2bdeb1efa86e0756e741292d625f91cb93d/datasette/views/table.py#L672-L704

With features like Facets in #255 I'm beginning to want to move more items into the template_data() - in the case of facets it's the suggested_facets array. This saves that feature from being calculated (involving several SQL queries) for the JSON case where it is unlikely to be used.

But... as an API user, I want to still optionally be able to access that information.

Solution: Add a ?_extra=suggested_facets&_extra=table_metadata argument which can be used to optionally request additional blocks to be added to the JSON API.

Then redefine as many of the current template_data() features as extra arguments instead, and teach Datasette to return certain extras by default when rendering templates.

This could allow the JSON representation to be slimmed down further (removing e.g. the table_definition and view_definition keys) while still making that information available to API users who need it.

e.g. first column on this page - cog menu has no option to facet.

https://datasette.io/content/tools

When looping through databases in the index.html template, I'd like to print the description of each database alongside its name. But it appears that isn't passed in from the view, unless I'm missing it. It would be great to have that.

Datasette currently uses the same table layout on the row pages as it does on the table pages:

https://datasette.io/content/pypi_packages?_sort=name&name__exact=datasette-column-inspect

https://datasette.io/content/pypi_packages/datasette-column-inspect

If you shrink down to mobile width you get this instead, on both of those pages:

I think that view, which I think of as the "card view", is plain better if you're looking at just a single row - and it (or a variant of it) should be the default presentation on the row page.

I just realized that when I ship Datasette 1.0 there may be automated deployments out there which could deploy the 1.0 version by accident, potentially breaking any customizations that aren't compatible with the 1.0 changes.

I can hopefully help avoid that by shipping one last entry in the 0.x series that ensures datasette publish pins to <1.0 when it installs Datasette itself.

I'd like to implement this by first extending the register_output_renderer() hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1096#issuecomment-732542285

I hadn't run into an issue with plugins like datasette-template-sql interfering with the existing context for other features before! Definitely not a good thing.

Originally posted by @simonw in https://github.com/simonw/datasette-write/issues/6#issuecomment-1379490596

Decision: as an initial fix I'm going to de-duplicate those keys by using tags__array etc - with a _2 on the end if that key is already used.

I'll open a separate issue to redesign this better for Datasette 1.0.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/625#issuecomment-996130862

In 49d6d2f7b0f6cb02e25022e1c9403811f1fa0a7c as part of #813 I removed the allow_sql setting - on the basis that users could disable the ability to execute custom SQL queries using the new permission system instead.

I don't think this was the right decision. Disabling custom SQL is an important security capability, and explaining how to do it using permissions is significantly more complex than letting people know they can add --setting allow_sql off.

So I want to bring that setting back - maybe with a different, better name - and have it modify the default for that option if the permissions system doesn't have an opinion.

That way people can still use the setting but then use permissions to allow specific signed-in users access to execute SQL.

Idea came from a conversation with @hydrosquall - what if a Datasette plugin could say "I'd like the JSON for a page to be included in a variable on the HTML page"?

datasette-cluster-map already needs this - the first thing it does when the page loads is fetch() a JSON representation of that same data.

This idea fits with my overall goals to unify the JSON and HTML context too.

Refs: - #1711

Split from: - #1931 - #1878

This is a bit tricky because the code needs to figure out what the primary keys are for an item, and whether or not rowid should be included.

Idea from here: - https://github.com/simonw/sqlite-utils/issues/516

Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions

This method allows multiple permissions to be checked at onced. It raises a datasette.Forbidden exception if any of the checks are denied before one of them is explicitly granted.

This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns True or not a single one of them returns False:

That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827

While working on this issue: - #1896

I realized that Datasette has error classes scattered around a fair bit, including some in the datasette.utils.asgi module for some reason.

I should clean these up.

This is a strong argument for extracting the Heroku support out to a plugin - it would allow this to be fixed with a plugin release without needing to push a full release of Datasette itself.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1905#issuecomment-1320678715

For datasette-copyable I want to show breadcrumbs that take database/instance permissions into account, so I'm removing {% block nav %} entirely and replacing it with this:

html+jinja {% block crumbs %} {{ crumbs.nav(request=request, database=database, table=table) }} {% endblock %}

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1901#issuecomment-1319588163

I should document this.

I was waiting for consensus to form around this (and kind-of hoping for trunk since I like the tree metaphor) and it looks like main is it.

I've seen convincing arguments against trunk too - it indicates that the branch has some special significance like in Subversion (where all branches come from trunk) when it doesn't. So main is better anyway.

Needed for the Datasette 1.0 release, so template authors can trust that Datasette is unlikely to break their templates.

This can drive the upgrade of the register_output_renderer hook to be able to handle streaming all rows in a large query.

This is a good idea, and one that needs to happen before Datasette 1.0:

While you are adding features, would you be future-proofing your APIs if you switched over some arguments over to keyword-only arguments or would that be too disruptive?

Thinking out loud:

async def render_template( self, templates, *, context=None, plugin_context=None, request=None, view_name=None ): Originally posted by @jefftriplett in https://github.com/simonw/datasette/issues/1817#issuecomment-1256781274

See https://github.com/simonw/datasette-publish-now/issues/9#issuecomment-618155764 - the --extra-options mechanism is in practice just used to set --config options in data that you publish, but that means you end up with pretty messy looking commands:

datasette publish my.db --extra-options="--config default_page_size:50 --config sql_time_limit_ms:3500"

A neater design would be to support --config as an option for datasette publish directly:

datasette publish my.db --config default_page_size:50 --config sql_time_limit_ms:3500

As part of the preparation for the 1.0 release, test all existing known plugins against the alpha.

For any that break, submit a PR suggesting they pin to a version <1.0 - and include a link to the documentation on how to upgrade the plugin for 1.0.

Tilde encoding introduced in #1657 means that database files with spaces in the name - e.g. the Apple Mail Envelope Index database - end up with URLs like this:

http://127.0.0.1:8001/Envelope~20Index

I think this would be prettier:

http://127.0.0.1:9933/Envelope+Index

Original title: Set content type for CSV so that browsers will attempt to download instead opening in the browser

Right now, if the user clicks on the CSV related to a <s>table or a</s> query, the response header for the content type is

"content-type: text/plain; charset=utf-8"

Most browsers will try to open a file with this content-type in the browser.

This is not what most people want to do, and lots of folks don't know that if they want to download the CSV and open it in the a spreadsheet program they next need to save the page through their browser.

It would be great if the response header could be something like

'Content-type: text/csv'); 'Content-disposition: attachment;filename=MyVerySpecial.csv');

which would lead browsers to open a download dialog.

I just created an alternative .fake() method because I wanted to fake the url_vars captured in the route as well: ```python from datasette.utils.asgi import Request class Request(Request):

@classmethod
def fake(cls, path_with_query_string, method="GET", scheme="http", url_vars=None):
    """Useful for constructing Request objects for tests"""
    path, _, query_string = path_with_query_string.partition("?")
    scope = {
        "http_version": "1.1",
        "method": method,
        "path": path,
        "raw_path": path_with_query_string.encode("latin-1"),
        "query_string": query_string.encode("latin-1"),
        "scheme": scheme,
        "type": "http",
    }
    if url_vars:
        scope["url_route"] = {
            "kwargs": url_vars
        }
    return cls(scope, None)

```

https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/utils/init.py#L181-L200

```python @contextmanager def sqlite_timelimit(conn, ms): deadline = time.perf_counter() + (ms / 1000) # n is the number of SQLite virtual machine instructions that will be # executed between each check. It's hard to know what to pick here. # After some experimentation, I've decided to go with 1000 by default and # 1 for time limits that are less than 50ms n = 1000 if ms < 50: n = 1

def handler():
    if time.perf_counter() >= deadline:
        return 1

conn.set_progress_handler(handler, n)
try:
    yield
finally:
    conn.set_progress_handler(None, n)

``` How often do I set a time limit of 50 or less? How much slower does it go thanks to this code?

Permission checks right now can express one of three opinions:

• False means "so not grant this permisson"
• True means "grant this permission"
• None means "I have no opinion"

But... there's also a concept of a "default" for a given permission check, which might be False or True.

I worry this is too complicated. Could this be simplified before 1.0? In particular the default concept.

See also: - #1676

While working on: - https://github.com/simonw/datasette/issues/1657 - https://github.com/simonw/datasette/issues/1439

It became very clear that the least maintainable part of Datasette at the moment is the way routing to the database, table and row views work - in particular the subclassing mechanism with BaseView and DataView, but also the complex variety of ways in which the URL routes capture different named regular expression groups.

Spotted this while working on: - #1677

https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/utils/init.py#L1005-L1021

Follow-on from: - #1675

Refs: - #1660

Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:

pandas.read_json("https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on", lines=True)

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1101#issuecomment-755128038

https://latest.datasette.io/.json

Currently: json { "_memory": { "name": "_memory", "hash": null, "color": "a6c7b9", "path": "/_memory", "tables_and_views_truncated": [], "tables_and_views_more": false, "tables_count": 0, "table_rows_sum": 0, "show_table_row_counts": false, "hidden_table_rows_sum": 0, "hidden_tables_count": 0, "views_count": 0, "private": false }, "fixtures": { "name": "fixtures", "hash": "645005884646eb941c89997fbd1c0dd6be517cb1b493df9816ae497c0c5afbaa", "color": "645005", "path": "/fixtures", "tables_and_views_truncated": [ { "name": "compound_three_primary_keys", "columns": [ "pk1", "pk2", "pk3", "content" ], "primary_keys": [ "pk1", "pk2", "pk3" ], "count": 1001, "hidden": false, "fts_table": null, "num_relationships_for_sorting": 0, "private": false }, As-of this issue the "path" key is confusing, it doesn't match what https://latest.datasette.io/-/databases returns:

• 1668

I think the way to get rid of most of the remaining complexity in DataView is to refactor how CSV stuff works - pulling it in line with other export factors and extracting the streaming mechanism. Opening a fresh issue for that.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1660#issuecomment-1073355032

Some issues came up in the new datasette-hashed-urls plugin relating to the way it renames databases on startup to achieve unique URLs that depend on the database SHA-256 content:

• https://github.com/simonw/datasette-hashed-urls/issues/10
• https://github.com/simonw/datasette-hashed-urls/issues/9
• https://github.com/simonw/datasette-hashed-urls/issues/8

All three of these could be addressed by making the "path" concept for a database (the /foo bit where it is served) work independently of the database's name, which would be used for default display and also as the alias when configuring cross-database aliases.

... highlights how inconsistent the way the capturing works is. Especially as_format which can be None or "" or .json or json or not used at all in the case of TableView.

https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670

Part of: - #1660

I ran into some bugs earlier with URL routing - having more robust testing around this (especially since they are defined using regular expressions) would be really useful.

• A utility function that resolves a path against a list of reflexes and returns the match
• Make the routes and regular expressions available from a private Datasette method
• Add tests that exercise them

Related: - #1660

Can be part of #870 - refactoring existing views to use register_routes().

I'm going to put the new check_permissions() method on BaseView as well. If I want that method to be available to plugins I can do so by turning that BaseView class into a documented API that plugins are encouraged to use themselves. Originally posted by @simonw in https://github.com/simonw/datasette/issues/832#issuecomment-651995453

They used to be the default until #418. Since making them optional I haven't felt the need to use them even once.

That suggests to me that they should be removed. I think their effect could be entirely handled by an ASGI wrapping plugin.

https://datasette.readthedocs.io/en/0.32/performance.html#hashed-url-mode

Tidy up from: - #1439

I've made a real mess of this. I'm going to revert Datasettemain back to the last commit that passed the tests and try this again in a branch.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1657#issuecomment-1068125636

The https://github.com/simonw/datasette-hashed-urls used a couple of currently undocumented features: - db.hash - Datasette(..., immutables=[...])

Tips here: https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/ - I think a PNG served at /favicon.ico is the best option, since safari doesn't support SVG yet.

Relevant code: https://github.com/simonw/datasette/blob/cb29119db9115b1f40de2fb45263ed77e3bfbb3e/datasette/app.py#L182-L183

I can reuse the icon for https://datasette.io/desktop

Original title: Consider placing example queries above the sql input?

Hi! Have been enjoying deploying ad hoc datasettes for collaborators to pick over!

I keep finding myself manually "fixing" the database.html template so that the "example queries" (canned queries) appear directly over the sql box? So they are sorta more a suggestion for collaborators who aren't inclined to write their own queries?

My sense is any time I go to the trouble of writing canned queries my users should see 'em?

(( I have also considered a client-side reactive-ish option where selecting a query just places the raw SQL in the box and doesn't execute it, but this seems to end up being an inconvenience, rather than a teaching tool. ))

To avoid another surprise like we got with 3.10: https://simonwillison.net/2021/Oct/9/finding-and-reporting-a-bug/

From a quick GitHub code search it looks like 3.11-dev should work: https://cs.github.com/urllib3/urllib3/blob/7bec77e81aa0a194c98381053225813f5347c9d2/.github/workflows/ci.yml#L60

Bug from: - #1620

% curl -s -I 'https://latest.datasette.io/-/patterns' | grep link link: https://latest.datasette.io/-/patterns.json; rel="alternate"; type="application/json+datasette"

% mkdir /tmp/data % cp ~/Dropbox/Development/datasette/fixtures.db /tmp/data % datasette /tmp/data/*.db /tmp/data/created.db --create -p 8852 ... INFO: Uvicorn running on http://127.0.0.1:8852 (Press CTRL+C to quit) ^CINFO: Shutting down % datasette /tmp/data/*.db /tmp/data/created.db --create -p 8852 ... INFO: 127.0.0.1:49533 - "GET / HTTP/1.1" 200 OK The first time I ran Datasette I got two databases - fixtures and created

BUT... when I ran Datasette the second time it looked like this:

This is the same result you get if you run:

datasette /tmp/data/fixtures.db /tmp/data/created.db /tmp/data/created.db

This is caused by this Datasette issue: - https://github.com/simonw/datasette/issues/509

So... either I teach Datasette to de-duplicate multiple identical file paths passed to the command, or I can't use /data/*.db in the Dockerfile here and I need to go back to other solutions for the challenge described in this comment: https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1031971831

Originally posted by @simonw in https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1032029874

It's now handled by a plugin instead: - #647 - https://github.com/simonw/datasette-hashed-urls/issues/3

https://github.com/simonw/datasette-hashed-urls

Sub-tasks:

• [x] Remove hashed URL mode implementation
• [x] Update documentation
• [x] Ensure --setting hash_urls 1 shows a useful message

I realized that one of the reasons I'm having trouble committing to nailing down the JSON API for 1.0 is that I don't use it much myself - I use the ?_shape=array one quite often, but I don't have any projects that are using the default, more fully-featured API.

As an experiment I built a Web Component for embedding Datasette tables on pages - https://github.com/simonw/datasette-table - and I think it's actually going to be a really useful tool for helping me dog food the v1.0 API design.

Datasette currently has surprising special behaviour for if a table name ends in .csv - which can happen when a tool like csvs-to-sqlite creates tables that match the filename that they were imported from.

https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv illustrates this behaviour: it links to .csv and .json that look like this:

• https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=json
• https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=csv&_size=max

Where normally Datasette would add the .csv or .json extension to the path component of the URL (as seen on other pages such as https://latest.datasette.io/fixtures/facet_cities) here the path_with_format() function notices that there is already a . in the path and instead adds ?_format=csv to the query string instead.

The problem with this mechanism is that it's pretty surprising. Anyone writing external code to Datasette who wants to get back the .csv or .json version giving the URL to a table page will need to know about and implement this behaviour themselves. That's likely to cause all kinds of bugs in the future.

Did a search on the issues here and didn't find anything related to what I want.

I want to have information that is on the database level of the JSON like title, source and source_url, and use it on the index page.

I tried some small tweaks on the python and html files, but failed to get that result.

Is there a way? Thanks!

https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/datasette/app.py#L297-L328

The new JSON API in a stable, documented form.

Split from #878. The current TableView class is by far the most complex part of Datasette, and the most difficult to work on: https://github.com/simonw/datasette/blob/0.59.2/datasette/views/table.py

In #878 I started exploring a new pattern for building views. In doing so it became clear that TableView is the first beast that I need to slay - if I can refactor that into something neat the pattern for building other views will emerge as a natural consequence.

I've been trying to build this as a register_routes() plugin, as originally suggested in #870 - though unfortunately it looks like those plugins can't replace existing Datasette default views at the moment, see #1517. [UPDATE: I was wrong about this, plugins can over-ride default views just fine]

I also know that I want to have a fully documented template context for table.html as a major step on the way to Datasette 1.0, see #1510.

All of this adds up to the TableView factor being a major project that will unblock a whole flurry of other things - so I'm going to work on that in this separate issue.

Refs #1439

The generated Dockerfile currently looks something like this: ```Dockerfile FROM python:3.8 COPY . /app WORKDIR /app

ENV DATASETTE_SECRET 'edab49cbc5d5f6f33238f54852037e3fee710821960b73edd2ce743454182ae2' RUN pip install -U datasette datasette-auth-passwords datasette-tiddlywiki datasette-graphql RUN datasette inspect fixtures.db other.db --inspect-file inspect-data.json ENV PORT 8080 EXPOSE 8080 CMD datasette serve --host 0.0.0.0 -i fixtures.db -i other.db --cors --inspect-file inspect-data.json --metadata metadata.json --create --port $PORT /data/*.db `` This is still on Python 3.8, and it generates a pretty large image compared to theDockerfile` used for https://hub.docker.com/datasetteproject/datasette - https://github.com/simonw/datasette/blob/0.60.2/Dockerfile

Here's the code that generates it: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/utils/init.py#L389-L400

Datasette's documentation is aimed at people who install and configure it.

What about end users of preconfigured and deployed Datasette instances?

Something that can be linked to from the Datasette UI would be really useful.

What I'm seeing

With default_cache_ttl = 86400, I see the following:

A table view returns Cache-control: max-age=86400:

A static asset returns no Cache-control header:

What I expected to see

I expected the static asset to return a Cache-control header indicating that this response can be cached.

Why this matters

I'm productionising a Datasette deployment right now and was looking into putting it behind a Varnish instance. I was surprised to see requests for static assets being served from Datasette rather than Varnish, this is what led me to look more closely at the response headers.

While Datasette serves those static assets pretty quickly, I don't see why Datasette should serve them. By their nature, static assets like images and JS files are very cacheable, so it should be easy to serve them from a cache like Varnish.

(Note that Varnish can easily be configured to override this header, enabling caching for static assets. But it would be better if this override was not necessary.)

Discussion

It seems clear to me that serving static assets without a Cache-control header is not ideal.

I see two options here:

A. Static assets use the same logic as table / SQL views to set the Cache-control header based on default_cache_ttl. B. An additional setting for static assets is introduced (default_static_cache_ttl, say).

One big advantage to this scheme is that redirecting old links to %2F pages (e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators) is easy - if you see a % in the raw_path, redirect to that page with the % replaced by -.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1439#issuecomment-1060044007

It would be nice if the --cors option accepted an origin regex to more securely allow secure local development.

As an example, Observable notebooks namespace every user's notebooks by their username and user content is served from username.observableusercontent.com, so you would set --cors-origin username.observableusercontent.com to restrict access to a local development Datasette instance to only your own notebooks, rather than exposing the data to any website that makes a request.

Thank you for all of your work on Datasette!

https://www.gaia-gis.it/gaia-sins/spatialite-sql-5.0.0.html#version has a whole bunch of interesting functions for things like freexl_version() and geos_version() and HasMathSQL() and suchlike.

These could be shown on the /-/versions page.

https://github.com/simonw/datasette-css-properties starts like this:

python from datasette import hookimpl from datasette.utils.asgi import Response from datasette.utils import escape_css_string, to_css_class escape_css_string and to_css_class are not documented, which means relying on them is risky since there's no promise that they won't change.

Would be good to figure out a policy on this, and maybe promote some of them to "documented" status.

Follows: - #1176

There are only a few classes that plugins need to import. It would be nice if these imports were as short and memorable as possible.

For example: python from datasette.app import Datasette from datasette.utils.asgi import Response Could both become: python from datasette import Datasette from datasette import Response

I tried running some parallel SQL queries using asyncio.gather() but the SQL that was executed didn't show up in the trace rendered by https://datasette.io/plugins/datasette-pretty-traces

I realized that was because traces are keyed against the current task ID, which changes when a sub-task is run using asyncio.gather or similar.

The faceting and suggest faceting queries are missing from this trace:

The reason they aren't showing up in the traces is that traces are stored just for the currently executing asyncio task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25

This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively "child tasks" of the current request, and hence should be tracked the same.

https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using asyncio.gather() to work around this kind of problem.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1518#issuecomment-999870993

Can use /database/-/... namespace from #296

Following: - #1533

I implemented it for tables and rows but I should have done queries as well.

Originally explored in https://github.com/simonw/datasette-notebook/issues/2#issuecomment-980789406 - I wanted an efficient way to scan a list of URLs and figure out which if any of those corresponded to Datasette tables, canned queries or SQL output that could be represented as a table on a page.

It looks like a neat way to do that is with Link: header like this:

Link: http://127.0.0.1:8058/fixtures/compound_three_primary_keys.json; rel="alternate"; type="application/datasette+json"

I can put a <link href=... in the page header too.

Original title: Decide when to drop support for Python 3.6

context_vars can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/

Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days?

Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1576#issuecomment-999878907

Nothing uses this plugin hook yet, so the design is not yet proven.

I'm going to build a real plugin against it and use that process to inform any design changes that may need to be made.

I'll add a warning about this to the documentation.

See https://github.com/simonw/datasette/issues/878#issuecomment-973554024_ - right now register_routes() can't replace default Datasette routes.

It would be neat if plugins could do this - especially if there was a neat documented way for them to then re-dispatch to the original route code after making some kind of modification.

Suggested by @frankieroberto in https://github.com/simonw/datasette/issues/782#issuecomment-782746755

Development Status :: 5 - Production/Stable

I created this view: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads-8dbda00/ads_with_targets

CREATE VIEW ads_with_targets as select ads.*, json_group_array(targets.name) as target_names from ads join ad_targets on ad_targets.ad_id = ads.id join targets on ad_targets.target_id = targets.id group by ad_targets.ad_id

When I try to apply faceting by array it appears to work at first: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads/ads_with_targets?_facet_array=target_names

But actually it's doing the wrong thing - the SQL for the facets uses rowid, but rowid is not present on views at all! These results are incorrect, and clicking to select a facet will fail to produce any rows: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads/ads_with_targets?_facet_array=target_names&target_names__arraycontains=people_who_match%3Ainterests%3AAfrican-American+Civil+Rights+Movement+%281954%E2%80%9468%29

Here's the SQL it should be using when you select a facet (note that it does not use a rowid):

https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select+*+from+ads_with_targets+where+id+in+%28%0D%0A++++++++++++select+ads_with_targets.id+from+ads_with_targets%2C+json_each%28ads_with_targets.target_names%29+j%0D%0A++++++++++++where+j.value+%3D+%3Ap0%0D%0A++++++++%29+limit+101&p0=people_who_match%3Ainterests%3ABlack+%28Color%29

So we need to do something a lot smarter here. I'm not sure what the fix will look like, or even if it's feasible given that views don't have a rowid to hook into so the JSON faceting SQL may have to be completely rewritten.

datasette publish cloudrun \ russian-ads.db \ --name json-view-facet-bug-demo \ --branch master \ --extra-options "--config sql_time_limit_ms:5000 --config facet_time_limit_ms:5000"

I need to perform a detailed review of the plugin interface - especially the plugin hooks like register_facet_classes() which I don't yet have complete confidence in.

Datasette ASGI #272 is a big part of it... but 1.0 will generally be an indicator that Datasette is a stable platform for developers to write plugins and custom templates against. So lots to think about.

If I was prone to over-thinking (which I am) I'd note that allow_facet and allow_download and allow_csv_stream are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent.

So maybe there's a future task where I introduce those as both permissions and metadata "allow_x" blocks, then rename the settings themselves to be called default_allow_facet and default_allow_download and default_allow_csv_stream.

If I was going to do that I should get it in before Datasette 1.0.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1409#issuecomment-890400425

docker run -it -v `pwd`:/mnt datasetteproject/datasette:latest /bin/bash root@0e1928cfdf79:/# cd /mnt root@0e1928cfdf79:/mnt# pip install -e .[test] root@0e1928cfdf79:/mnt# pytest I get one failure!

It was for test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3] ``` def test_searchable(app_client, path, expected_rows): response = app_client.get(path)

  assert expected_rows == response.json["rows"]

E AssertionError: assert [[1, 'barry c...sel', 'puma']] == [] E Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther'] E Full diff: E + [] E - [[1, 'barry cat', 'terry dog', 'panther'], E - [2, 'terry dog', 'sara weasel', 'puma']] ```

Originally posted by @simonw in https://github.com/simonw/datasette/issues/695#issuecomment-595614469

I want to theme https://datasette.io/ so that when you visit https://datasette.io/content (the Datasette UI part of it) the navigation from the parent site is used.

I tried dropping in a base.html template like this:

```html {% extends "page_base.html" %}

{% block base_extra_head %}

<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> {% for url in extra_css_urls %} <link rel="stylesheet" href="{{ url.url }}"{% if url.sri %} integrity="{{ url.sri }}" crossorigin="anonymous"{% endif %}> {% endfor %} {% for url in extra_js_urls %} <script src="{{ url.url }}"{% if url.sri %} integrity="{{ url.sri }}" crossorigin="anonymous"{% endif %}></script> {% endfor %} {% block extra_head %}{% endblock %} {% endblock %}

{% block extra_body_end %} {% include "_close_open_menus.html" %}

{% for body_script in body_scripts %} <script>{{ body_script }}</script> {% endfor %} {% endblock %} ``` But this resulted in pages looking like this:

Note that the cog menu is broken and the filter UI is unstyled. To get these working correctly I would need to copy over a whole lot of Datasette's default CSS - and that means that when Datasette changes in the future those pages could break in subtle ways.

My httpx pull request adding raw_path support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.

I'm going to mark these tests as xfail so I can land this change - I'll remove that once an httpx release comes out that I can use to get the tests passing.

Originally posted by @simonw in https://github.com/simonw/datasette/pull/1000#issuecomment-706263157

For consistency with /_internal - and because then we don't need to escape the : characters.

This change would need to be in before Datasette 1.0. I could land it earlier and set up redirects from the old URLs though.

Quite a few of the plugin hooks make a datasette”instance of the Datasette class available to the plugins, so that they can look up configuration settings and execute database queries.

This means it should provide a documented, stable API so that plugin authors can rely on it.

I want Datasette to have its own internal metadata about connected tables, to power features like a paginated searchable homepage in #461. I want this to be a SQLite table.

This could also be part of the directory scanning mechanism prototyped in #672 - where Datasette can be set to continually scan a directory for new database files that it can serve.

Also relevant to the Datasette Library concept in #417.

e.g. I should be able to do this:

datasette App/data.db Other_App/data.db

This currently errors because you can't have two databases taking the /data URL path.

Instead, how about in this particular case assigning the second database /data-1?

Split out from #460 - in order to support large numbers of connected databases the homepage needs to be paginated.

I added a deprecation warning to this in #992.

I've seen attempted downloads of large files fail after about ten seconds.

In #998 I made a change that affected the table page but didn't affect the query page because I incorrectly assumed they shared rendering logic.

Some databases have a LOT of tables. Now that we often calculate table row counts dynamically we could really speed things up by paginating the database index page, e.g. http://fivethirtyeight-datasette.herokuapp.com/fivethirtyeight

If we're paginating, having a filter-search-for-table widget (similar to the search-for-database widget I'm planning for the homepage) would make sense.

Related: pagination for homepage #461 and Datasette Library #417

Several plugin hooks now expect plugins to return data in a specific shape - notably the new output format hook and the custom facet hook.

These use Python dictionaries right now but that's quite error prone: it would be good to have a mechanism that supported a more structured format.

Full list of current hooks is here: https://datasette.readthedocs.io/en/latest/plugins.html#plugin-hooks

I'm having a lot of trouble copying and pasting from the codemirror editor on my iPhone.