issues

It is increasingly being useful configuration options, when it started out as purely metadata.

Could cause confusion with the --config mechanism though - maybe that should be called "settings" instead?

I think this has been a bug for a long time.

https://latest.datasette.io/ currently shows:

Those colors are based on a hash of the database name. But when you click through to https://latest.datasette.io/fixtures

It's red on all sub-pages too.

In 49d6d2f7b0f6cb02e25022e1c9403811f1fa0a7c as part of #813 I removed the allow_sql setting - on the basis that users could disable the ability to execute custom SQL queries using the new permission system instead.

I don't think this was the right decision. Disabling custom SQL is an important security capability, and explaining how to do it using permissions is significantly more complex than letting people know they can add --setting allow_sql off.

So I want to bring that setting back - maybe with a different, better name - and have it modify the default for that option if the permissions system doesn't have an opinion.

That way people can still use the setting but then use permissions to allow specific signed-in users access to execute SQL.

I was waiting for consensus to form around this (and kind-of hoping for trunk since I like the tree metaphor) and it looks like main is it.

I've seen convincing arguments against trunk too - it indicates that the branch has some special significance like in Subversion (where all branches come from trunk) when it doesn't. So main is better anyway.

Needed for the Datasette 1.0 release, so template authors can trust that Datasette is unlikely to break their templates.

Tilde encoding introduced in #1657 means that database files with spaces in the name - e.g. the Apple Mail Envelope Index database - end up with URLs like this:

http://127.0.0.1:8001/Envelope~20Index

I think this would be prettier:

http://127.0.0.1:9933/Envelope+Index

I just created an alternative .fake() method because I wanted to fake the url_vars captured in the route as well: ```python from datasette.utils.asgi import Request class Request(Request):

@classmethod
def fake(cls, path_with_query_string, method="GET", scheme="http", url_vars=None):
    """Useful for constructing Request objects for tests"""
    path, _, query_string = path_with_query_string.partition("?")
    scope = {
        "http_version": "1.1",
        "method": method,
        "path": path,
        "raw_path": path_with_query_string.encode("latin-1"),
        "query_string": query_string.encode("latin-1"),
        "scheme": scheme,
        "type": "http",
    }
    if url_vars:
        scope["url_route"] = {
            "kwargs": url_vars
        }
    return cls(scope, None)

```

https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/utils/init.py#L181-L200

```python @contextmanager def sqlite_timelimit(conn, ms): deadline = time.perf_counter() + (ms / 1000) # n is the number of SQLite virtual machine instructions that will be # executed between each check. It's hard to know what to pick here. # After some experimentation, I've decided to go with 1000 by default and # 1 for time limits that are less than 50ms n = 1000 if ms < 50: n = 1

def handler():
    if time.perf_counter() >= deadline:
        return 1

conn.set_progress_handler(handler, n)
try:
    yield
finally:
    conn.set_progress_handler(None, n)

``` How often do I set a time limit of 50 or less? How much slower does it go thanks to this code?

While working on: - https://github.com/simonw/datasette/issues/1657 - https://github.com/simonw/datasette/issues/1439

It became very clear that the least maintainable part of Datasette at the moment is the way routing to the database, table and row views work - in particular the subclassing mechanism with BaseView and DataView, but also the complex variety of ways in which the URL routes capture different named regular expression groups.

Spotted this while working on: - #1677

https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/utils/init.py#L1005-L1021

Follow-on from: - #1675

Refs: - #1660

Some issues came up in the new datasette-hashed-urls plugin relating to the way it renames databases on startup to achieve unique URLs that depend on the database SHA-256 content:

• https://github.com/simonw/datasette-hashed-urls/issues/10
• https://github.com/simonw/datasette-hashed-urls/issues/9
• https://github.com/simonw/datasette-hashed-urls/issues/8

All three of these could be addressed by making the "path" concept for a database (the /foo bit where it is served) work independently of the database's name, which would be used for default display and also as the alias when configuring cross-database aliases.

... highlights how inconsistent the way the capturing works is. Especially as_format which can be None or "" or .json or json or not used at all in the case of TableView.

https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670

Part of: - #1660

I ran into some bugs earlier with URL routing - having more robust testing around this (especially since they are defined using regular expressions) would be really useful.

• A utility function that resolves a path against a list of reflexes and returns the match
• Make the routes and regular expressions available from a private Datasette method
• Add tests that exercise them

Related: - #1660

They used to be the default until #418. Since making them optional I haven't felt the need to use them even once.

That suggests to me that they should be removed. I think their effect could be entirely handled by an ASGI wrapping plugin.

https://datasette.readthedocs.io/en/0.32/performance.html#hashed-url-mode

Tidy up from: - #1439

I've made a real mess of this. I'm going to revert Datasettemain back to the last commit that passed the tests and try this again in a branch.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1657#issuecomment-1068125636

The https://github.com/simonw/datasette-hashed-urls used a couple of currently undocumented features: - db.hash - Datasette(..., immutables=[...])

Tips here: https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/ - I think a PNG served at /favicon.ico is the best option, since safari doesn't support SVG yet.

Relevant code: https://github.com/simonw/datasette/blob/cb29119db9115b1f40de2fb45263ed77e3bfbb3e/datasette/app.py#L182-L183

I can reuse the icon for https://datasette.io/desktop

Original title: Consider placing example queries above the sql input?

Hi! Have been enjoying deploying ad hoc datasettes for collaborators to pick over!

I keep finding myself manually "fixing" the database.html template so that the "example queries" (canned queries) appear directly over the sql box? So they are sorta more a suggestion for collaborators who aren't inclined to write their own queries?

My sense is any time I go to the trouble of writing canned queries my users should see 'em?

(( I have also considered a client-side reactive-ish option where selecting a query just places the raw SQL in the box and doesn't execute it, but this seems to end up being an inconvenience, rather than a teaching tool. ))

To avoid another surprise like we got with 3.10: https://simonwillison.net/2021/Oct/9/finding-and-reporting-a-bug/

From a quick GitHub code search it looks like 3.11-dev should work: https://cs.github.com/urllib3/urllib3/blob/7bec77e81aa0a194c98381053225813f5347c9d2/.github/workflows/ci.yml#L60

Bug from: - #1620

% curl -s -I 'https://latest.datasette.io/-/patterns' | grep link link: https://latest.datasette.io/-/patterns.json; rel="alternate"; type="application/json+datasette"

% mkdir /tmp/data % cp ~/Dropbox/Development/datasette/fixtures.db /tmp/data % datasette /tmp/data/*.db /tmp/data/created.db --create -p 8852 ... INFO: Uvicorn running on http://127.0.0.1:8852 (Press CTRL+C to quit) ^CINFO: Shutting down % datasette /tmp/data/*.db /tmp/data/created.db --create -p 8852 ... INFO: 127.0.0.1:49533 - "GET / HTTP/1.1" 200 OK The first time I ran Datasette I got two databases - fixtures and created

BUT... when I ran Datasette the second time it looked like this:

This is the same result you get if you run:

datasette /tmp/data/fixtures.db /tmp/data/created.db /tmp/data/created.db

This is caused by this Datasette issue: - https://github.com/simonw/datasette/issues/509

So... either I teach Datasette to de-duplicate multiple identical file paths passed to the command, or I can't use /data/*.db in the Dockerfile here and I need to go back to other solutions for the challenge described in this comment: https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1031971831

Originally posted by @simonw in https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1032029874

It's now handled by a plugin instead: - #647 - https://github.com/simonw/datasette-hashed-urls/issues/3

https://github.com/simonw/datasette-hashed-urls

Sub-tasks:

• [x] Remove hashed URL mode implementation
• [x] Update documentation
• [x] Ensure --setting hash_urls 1 shows a useful message

Datasette currently has surprising special behaviour for if a table name ends in .csv - which can happen when a tool like csvs-to-sqlite creates tables that match the filename that they were imported from.

https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv illustrates this behaviour: it links to .csv and .json that look like this:

• https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=json
• https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=csv&_size=max

Where normally Datasette would add the .csv or .json extension to the path component of the URL (as seen on other pages such as https://latest.datasette.io/fixtures/facet_cities) here the path_with_format() function notices that there is already a . in the path and instead adds ?_format=csv to the query string instead.

The problem with this mechanism is that it's pretty surprising. Anyone writing external code to Datasette who wants to get back the .csv or .json version giving the URL to a table page will need to know about and implement this behaviour themselves. That's likely to cause all kinds of bugs in the future.

Refs #1439

One big advantage to this scheme is that redirecting old links to %2F pages (e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators) is easy - if you see a % in the raw_path, redirect to that page with the % replaced by -.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1439#issuecomment-1060044007

https://www.gaia-gis.it/gaia-sins/spatialite-sql-5.0.0.html#version has a whole bunch of interesting functions for things like freexl_version() and geos_version() and HasMathSQL() and suchlike.

These could be shown on the /-/versions page.

https://github.com/simonw/datasette-css-properties starts like this:

python from datasette import hookimpl from datasette.utils.asgi import Response from datasette.utils import escape_css_string, to_css_class escape_css_string and to_css_class are not documented, which means relying on them is risky since there's no promise that they won't change.

Would be good to figure out a policy on this, and maybe promote some of them to "documented" status.

There are only a few classes that plugins need to import. It would be nice if these imports were as short and memorable as possible.

For example: python from datasette.app import Datasette from datasette.utils.asgi import Response Could both become: python from datasette import Datasette from datasette import Response

I tried running some parallel SQL queries using asyncio.gather() but the SQL that was executed didn't show up in the trace rendered by https://datasette.io/plugins/datasette-pretty-traces

I realized that was because traces are keyed against the current task ID, which changes when a sub-task is run using asyncio.gather or similar.

The faceting and suggest faceting queries are missing from this trace:

The reason they aren't showing up in the traces is that traces are stored just for the currently executing asyncio task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25

This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively "child tasks" of the current request, and hence should be tracked the same.

https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using asyncio.gather() to work around this kind of problem.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1518#issuecomment-999870993

Following: - #1533

I implemented it for tables and rows but I should have done queries as well.

Originally explored in https://github.com/simonw/datasette-notebook/issues/2#issuecomment-980789406 - I wanted an efficient way to scan a list of URLs and figure out which if any of those corresponded to Datasette tables, canned queries or SQL output that could be represented as a table on a page.

It looks like a neat way to do that is with Link: header like this:

Link: http://127.0.0.1:8058/fixtures/compound_three_primary_keys.json; rel="alternate"; type="application/datasette+json"

I can put a <link href=... in the page header too.

Original title: Decide when to drop support for Python 3.6

context_vars can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/

Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days?

Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1576#issuecomment-999878907

See https://github.com/simonw/datasette/issues/878#issuecomment-973554024_ - right now register_routes() can't replace default Datasette routes.

It would be neat if plugins could do this - especially if there was a neat documented way for them to then re-dispatch to the original route code after making some kind of modification.

I created this view: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads-8dbda00/ads_with_targets

CREATE VIEW ads_with_targets as select ads.*, json_group_array(targets.name) as target_names from ads join ad_targets on ad_targets.ad_id = ads.id join targets on ad_targets.target_id = targets.id group by ad_targets.ad_id

When I try to apply faceting by array it appears to work at first: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads/ads_with_targets?_facet_array=target_names

But actually it's doing the wrong thing - the SQL for the facets uses rowid, but rowid is not present on views at all! These results are incorrect, and clicking to select a facet will fail to produce any rows: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads/ads_with_targets?_facet_array=target_names&target_names__arraycontains=people_who_match%3Ainterests%3AAfrican-American+Civil+Rights+Movement+%281954%E2%80%9468%29

Here's the SQL it should be using when you select a facet (note that it does not use a rowid):

https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select+*+from+ads_with_targets+where+id+in+%28%0D%0A++++++++++++select+ads_with_targets.id+from+ads_with_targets%2C+json_each%28ads_with_targets.target_names%29+j%0D%0A++++++++++++where+j.value+%3D+%3Ap0%0D%0A++++++++%29+limit+101&p0=people_who_match%3Ainterests%3ABlack+%28Color%29

So we need to do something a lot smarter here. I'm not sure what the fix will look like, or even if it's feasible given that views don't have a rowid to hook into so the JSON faceting SQL may have to be completely rewritten.

datasette publish cloudrun \ russian-ads.db \ --name json-view-facet-bug-demo \ --branch master \ --extra-options "--config sql_time_limit_ms:5000 --config facet_time_limit_ms:5000"

Datasette ASGI #272 is a big part of it... but 1.0 will generally be an indicator that Datasette is a stable platform for developers to write plugins and custom templates against. So lots to think about.

docker run -it -v `pwd`:/mnt datasetteproject/datasette:latest /bin/bash root@0e1928cfdf79:/# cd /mnt root@0e1928cfdf79:/mnt# pip install -e .[test] root@0e1928cfdf79:/mnt# pytest I get one failure!

It was for test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3] ``` def test_searchable(app_client, path, expected_rows): response = app_client.get(path)

  assert expected_rows == response.json["rows"]

E AssertionError: assert [[1, 'barry c...sel', 'puma']] == [] E Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther'] E Full diff: E + [] E - [[1, 'barry cat', 'terry dog', 'panther'], E - [2, 'terry dog', 'sara weasel', 'puma']] ```

Originally posted by @simonw in https://github.com/simonw/datasette/issues/695#issuecomment-595614469

My httpx pull request adding raw_path support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.

I'm going to mark these tests as xfail so I can land this change - I'll remove that once an httpx release comes out that I can use to get the tests passing.

Originally posted by @simonw in https://github.com/simonw/datasette/pull/1000#issuecomment-706263157

For consistency with /_internal - and because then we don't need to escape the : characters.

This change would need to be in before Datasette 1.0. I could land it earlier and set up redirects from the old URLs though.

Quite a few of the plugin hooks make a datasette”instance of the Datasette class available to the plugins, so that they can look up configuration settings and execute database queries.

This means it should provide a documented, stable API so that plugin authors can rely on it.

I want Datasette to have its own internal metadata about connected tables, to power features like a paginated searchable homepage in #461. I want this to be a SQLite table.

This could also be part of the directory scanning mechanism prototyped in #672 - where Datasette can be set to continually scan a directory for new database files that it can serve.

Also relevant to the Datasette Library concept in #417.

e.g. I should be able to do this:

datasette App/data.db Other_App/data.db

This currently errors because you can't have two databases taking the /data URL path.

Instead, how about in this particular case assigning the second database /data-1?

I've seen attempted downloads of large files fail after about ten seconds.

Some databases have a LOT of tables. Now that we often calculate table row counts dynamically we could really speed things up by paginating the database index page, e.g. http://fivethirtyeight-datasette.herokuapp.com/fivethirtyeight

If we're paginating, having a filter-search-for-table widget (similar to the search-for-database widget I'm planning for the homepage) would make sense.

Related: pagination for homepage #461 and Datasette Library #417

Several plugin hooks now expect plugins to return data in a specific shape - notably the new output format hook and the custom facet hook.

These use Python dictionaries right now but that's quite error prone: it would be good to have a mechanism that supported a more structured format.

Full list of current hooks is here: https://datasette.readthedocs.io/en/latest/plugins.html#plugin-hooks

I'm having a lot of trouble copying and pasting from the codemirror editor on my iPhone.

Steps to reproduce: visit https://2a4b892.datasette.io/fixtures/roadside_attractions?_facet_m2m=attraction_characteristic and click "Apply"

Result is a 500: no such column: attraction_characteristic

The error occurs because of this hidden HTML input:

<input type="hidden" name="_facet" value="attraction_characteristic">

This should be:

<input type="hidden" name="_facet_m2m" value="attraction_characteristic">

See this conversation: https://twitter.com/psychemedia/status/1199707352540368896

Idea: an "allow_sql" key with a SQL query that gets passed the actor JSON as :actor and can extract the relevant keys from it and return 1 or 0.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/698#issuecomment-639787304

See also #800

internals.rst will be landing as part of #683

Refs #698. Replaces #703

Still todo:

• [x] Unit tests
• ~~Figure out .json mode~~
• [x] Flash message solution
• ~~CSRF protection~~
• [x] Better error message display on errors
• [x] Documentation
• ~~Maybe widgets?~~ I'll do these later

e.g. json { "name": "qs_actor.py", "static": false, "templates": false, "version": null, "hooks": [ "actor_from_request" ] }

See #682

• [x] Come up with design for .execute_write() and .execute_write_fn()
• [x] Build some quick demo plugins to exercise the design
• [x] Write some unit tests
• [x] Write the documentation

In #683 I started a new section of internals documentation covering the Database class: https://datasette.readthedocs.io/en/latest/internals.html#database-class

I decided not to document .execute() and .execute_against_connection_in_thread() yet because I'm not 100% happy with their API design yet.

Since that object is passed to the extra_template_vars hooks AND the classes registered by register_facet_classes it should be part of the documented interface on https://datasette.readthedocs.io/en/stable/internals.html

I could also start passing it to the register_output_renderer callback.

request.raw_args is not documented, and I'd like to remove it entirely. Originally posted by @simonw in https://github.com/simonw/datasette/issues/706#issuecomment-634975252

I use it in a few places in other projects though, so I'll have to fix those first: https://github.com/search?q=user%3Asimonw+raw_args&type=Code

Sometimes a plugin will need to render a template for some custom UI. We need a documented API for doing this, which ensures that everything will work correctly if you extend base.html etc.

See also #576. This could be a .render() method on the Datasette class, but that feels a bit weird - should that class also take responsibility for rendering?

Most of the code here was fleshed out in comments on #272 (Port Datasette to ASGI) - this pull request will track the final pieces:

• [x] Update test harness to more correctly simulate the raw_path issue
• [x] Use raw_path so table names containing / can work correctly
• [x] Bug: JSON not served with correct content-type
• [x] Get ?_trace=1 working again
• [x] Replacement for @app.listener("before_server_start")
• [x] Bug: /fixtures/table%2Fwith%2Fslashes.csv?_format=json downloads as CSV
• [x] Replace Sanic request and response objects with my own classes, so I can remove Sanic dependency
• [x] Final code tidy-up before merging to master

Datasette doesn't take much advantage of Sanic, and I'm increasingly having to work around parts of it because of idiosyncrasies that are specific to Datasette - caring about the exact order of querystring arguments for example.

Since Datasette is GET-only our needs from a web framework are actually pretty slim.

This becomes more important as I expand the plugins #14 framework. Am I sure I want the plugin ecosystem to depend on a Sanic if I might move away from it in the future?

If Datasette wasn't all about async/await I would use WSGI, but today it makes more sense to use ASGI. I'd like to be confident that switching to ASGI would still give me the excellent performance that Sanic provides.

https://github.com/django/asgiref/blob/master/specs/asgi.rst