The PR moves the following fields from metadata.yaml to datasette.yaml:

permissions allow allow_sql queries extra_css_urls extra_js_urls

This is a significant breaking change that users will need to upgrade their metadata.yaml files for. But the format/locations are similar to the previous version, so it shouldn't be too difficult to upgrade.

One note: I'm still working on the Configuration docs, specifically the "reference" section. Though it's pretty small, the rest of read to review

Part of #2093

In #2149 , we ported over "settings.json" into the new datasette.yaml config file, with a top-level "settings" key. This PR ports over plugin configuration into top-level "plugins" key, as well as nested database/table plugin config.

From now on, no plugin-related configuration is allowed in metadata.yaml, and must be in datasette.yaml in this new format. This is a pretty significant breaking change. Thankfully, you should be able to copy-paste your legacy plugin key/values into the new datasette.yaml format.

An example of what datasette.yaml would look like with this new plugin config:

```yaml

plugins: datasette-my-plugin: config_key: value

databases: fixtures: plugins: datasette-my-plugin: config_key: fixtures-db-value tables: students: plugins: datasette-my-plugin: config_key: fixtures-students-table-value

```

As an additional benefit, this now works with the new -s flag:

bash datasette --memory -s 'plugins.datasette-my-plugin.config_key' new_value

Marked as a "Draft" right now until I add better documentation. We also should have a plan for the next alpha release to document and publicize this change, especially for plugin authors (since their docs will have to change to say datasette.yaml instead of metadata.yaml

:books: Documentation preview :books:: https://datasette--2183.org.readthedocs.build/en/2183/

refs #2157

This PR adds a new --internal option to datasette serve. If provided, it is the path to a persistent internal database that Datasette core and Datasette plugins can use to store data, as discussed in the proposal issue.

This PR also removes and deprecates the previous in-memory _internal database. Those tables now appear in the internal database, with core_ prefixes (ex tables in _internal is now core_tables in internal).

A note on the new core_ tables

However, one important notes about those new core_ tables: If a --internal DB is passed in, that means those core_ tables will persist across multiple Datasette instances. This wasn't the case before, since _internal was always an in-memory database created from scratch.

I tried to put those core_ tables as TEMP tables - after all, there's always one 1 internal DB connection at a time, so I figured it would work. But, since we use the Database() wrapper for the internal DB, it has two separate connections: a default read-only connection and a write connection that is created when a write operation occurs. Which meant the TEMP tables would be created by the write connection, but not available in the read-only connection.

So I had a brillant idea: Attach an in-memory named database with cache=shared, and create those tables there!

sql ATTACH DATABASE 'file:datasette_internal_core?mode=memory&cache=shared' AS core;

We'd run this on both the read-only connection and the write-only connection. That way, those tables would stay in memory, they'd communicate with the cache=shared feature, and we'd be good to go.

However, I couldn't find an easy way to run a ATTACH DATABASE command on the read-only query.

Using Database() as a wrapper for the internal DB is pretty limiting - it's meant for Datasette "data" databases, where we want multiple readers and possibly 1 write connection at a time. But the internal database doesn't really require that kind of support - I think we could get away with a single read/write connection, but it seemed like too big of a rabbithole to go through now.

:books: Documentation preview :books:: https://datasette--2162.org.readthedocs.build/en/2162/

See: - https://github.com/simonw/datasette-configure-fts/issues/14 - https://github.com/simonw/datasette-auth-tokens/issues/12

I just noticed that https://latest.datasette.io/fixtures/roadside_attraction_characteristics.json?_labels=on&_size=1 returns: json { "ok": true, "next": "1", "rows": [ { "rowid": 1, "attraction_id": { "value": 1, "label": "The Mystery Spot" }, "characteristic_id": { "value": 2, "label": "Paranormal" } } ], "truncated": false } But https://latest.datasette.io/fixtures.json?sql=select+rowid%2C+attraction_id%2C+characteristic_id+from+roadside_attraction_characteristics+order+by+rowid+limit+1 returns: json { "rows": [ { "rowid": 1, "attraction_id": 1, "characteristic_id": 2 } ], "columns": [ "rowid", "attraction_id", "characteristic_id" ], "ok": true, "truncated": false } The columns key in the query response is inconsistent with the table response.

This will be the base that the remaining work builds on top of. Refs: - #2109

There are still a bunch of bugs in CodeMirror 5 that affect various mobile browsers - see Datasette Discord report here: https://discord.com/channels/823971286308356157/823971286941302908/1013878624992108645

https://user-images.githubusercontent.com/9599/187349269-7b7c0c8c-3894-4810-82f0-de7c1eb940b3.mp4

https://github.com/mitsuhiko/rye

I tried this:

rye install datasette

But now:

% ~/.rye/shims/datasette Traceback (most recent call last): File "/Users/simon/.rye/shims/datasette", line 5, in <module> from datasette.cli import cli File "/Users/simon/.rye/tools/datasette/lib/python3.11/site-packages/datasette/cli.py", line 17, in <module> from .app import ( File "/Users/simon/.rye/tools/datasette/lib/python3.11/site-packages/datasette/app.py", line 14, in <module> import pkg_resources ModuleNotFoundError: No module named 'pkg_resources' I think that's because setuptools is not included in Rye.

Need to stop linking to it from the docs.

I'll link to https://www.python.org/about/gettingstarted/ instead.

This should be pretty easy because Uvicorn supports them already. Need a good mechanism for testing it - https://pypi.org/project/trustme/ looks ideal.

I installed datasette==1.0a1.

When I go to http://127.0.0.1:8001/-/api I have a link: Use this tool to try out the [Datasette API](https://docs.datasette.io/en/1.0a1/json_api.html). but that documentation page does not exist.

I'm not sure where it has to be fixed, should it link to the stable page https://docs.datasette.io/en/stable/json_api.html , the latest one https://docs.datasette.io/en/latest/json_api.html#the-json-write-api or would it be more appropriated to deploy documentation for the 1.0a1 version?

Equivalent to https://sqlite-utils.datasette.io/en/stable/cli.html#insert-replacing-data

This release will mainly help preview the new Datasette write API: - #1850

API design: POST /db/table/row-pks/-/delete Or... DELETE /db/table/row-pks/-/delete I'm just going to do POST for the moment, like I did here: - #1874

Permission: delete-row

Still needed:

• [ ] Tests for rowid tables
• [ ] Tests for compound primary keys

Reported on Discord: https://discord.com/channels/823971286308356157/823971286941302908/1042814317118115901

``` -----> Building on the Heroku-22 stack -----> Determining which buildpack to use for this app -----> Python app detected -----> Using Python version specified in runtime.txt ! Requested runtime 'python-3.8.10' is not available for this stack (heroku-22). ! For supported versions, see: https://devcenter.heroku.com/articles/python-support ! Push rejected, failed to compile Python app.

! Push failed ▸ Build failed ```

In trying to write this I realize that there's a lot of duplicated code with delete row, specifically around resolving the incoming URL into a row (or a database or a table).

Since this is so common, I think it's worth extracting the logic out first.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1863#issuecomment-1317755263

After the upgrade to 6 (#1893) I noticed this. I think it's because we're doing overflow:hidden to accomplish the CSS resizer.

When there's a single line of SQL there's a gap below that line where clicking doesn't do anything. It should focus at the end of the line.

POST /db/table/-/drop

Require drop-table permission.

It's currently possible to use /-/create-token to create a token that lasts forever.

Some administrators may wish to have a maximum expiry instead. I should support that with a setting.

Datasette Lite: https://lite.datasette.io/?sql=https://gist.githubusercontent.com/simonw/1f8a91123ccefd8844187225b1832d7a/raw/5069075b86aa79358fbab3d4482d1d269077d632/recipes.sql#/data?sql=select+id%2C+name%2C+ingredients%2C+%28%0A++select+json_group_array%28value%29+from+json_each%28ingredients%29%0A++where+value+in+%28select+value+from+json_each%28%3Ap0%29%29%0A%29+as+matching_ingredients%0Afrom+recipes%0Awhere+json_array_length%28matching_ingredients%29+%3E+0%0Aorder+by+json_array_length%28matching_ingredients%29+desc&p0=%5B%22sugar%22%2C+%22cheese%22%5D

I guess this issue is not very important and probably rare.

To reproduce: * create and populate a db named off.db * in the yaml file, add any kind of information below databases:\n off: * the data are not taken into account (because "off" is interpreted as "false")

YAML file: ```yaml title: Some title description_html: |-

This is an experiment.

databases: off: tables: products_from_owners: title: products_from_owners* description_html: |-

Description

The result for http://xxxx.xxx/-/metadata gives: json { "title": "Some title", "description_html": "<p>This is an experiment.</p>", "databases": { "false": { "tables": { "products_from_owners": { "title": "products_from_owners*", "description_html": "<p>Description</p>" } } } } } => see the "false" instead of "off".

For example: https://latest.datasette.io/fixtures/sortable?_sort_desc=sortable&_col=sortable_with_nulls

That's ?_sort_desc=sortable&_col=sortable_with_nulls

Refs: - #1733

Need something in the test suite such that if Datasette breaks against Pyodide in the future we hear about it.

I'm thinking this is an opportunity to use shot-scraper javascript.

Relates to #1533 - and to the work I've been doing on the https://github.com/simonw/datasette-table Web Component.

It would be useful to support users pasting in a URL to a Datasette table or query without first having to add the .json extension themselves - since then other systems could hit that URL with Accept: application/json to get back the JSON representation without first needing to read the Link: header from #1533 to figure out what the URL to that JSON is.

(There is weird logic deep in Datasette that says that you add .json to the path UNLESS the table name itself ends with .json, in which case you add ?_format=json - this is super-confusing).

[Update: I removed that confusing feature here: https://simonwillison.net/2022/Mar/19/weeknotes/]

Is it possible to configure the sql_time_limit_ms beyond 60 seconds? It seems queries are still timing out at 60 seconds when sql_time_limit_ms is set to 180000. We have a very large data set and often encounter timeouts when testing new queries from the datasette UI. We are optimizing our database as much as we can, but still may require more than 60 seconds for complex queries.

Errors like this one:

https://github.com/simonw/datasette/pull/927/checks?check_run_id=973559696

2020-08-11T23:36:39.8801334Z =================================== FAILURES =================================== 2020-08-11T23:36:39.8802411Z _________________________________ test_install _________________________________ 2020-08-11T23:36:39.8803242Z 2020-08-11T23:36:39.8804935Z thing = <module 'pip._internal.cli' from '/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/pip/_internal/cli/__init__.py'> 2020-08-11T23:36:39.8806663Z comp = 'main', import_path = 'pip._internal.cli.main' 2020-08-11T23:36:39.8807696Z 2020-08-11T23:36:39.8808728Z def _dot_lookup(thing, comp, import_path): 2020-08-11T23:36:39.8810573Z try: 2020-08-11T23:36:39.8812262Z > return getattr(thing, comp) 2020-08-11T23:36:39.8817136Z E AttributeError: module 'pip._internal.cli' has no attribute 'main' 2020-08-11T23:36:39.8843043Z 2020-08-11T23:36:39.8855951Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/unittest/mock.py:1215: AttributeError 2020-08-11T23:36:39.8873372Z 2020-08-11T23:36:39.8877803Z During handling of the above exception, another exception occurred: 2020-08-11T23:36:39.8906532Z 2020-08-11T23:36:39.8925767Z def get_src_prefix(): 2020-08-11T23:36:39.8928277Z # type: () -> str 2020-08-11T23:36:39.8930068Z if running_under_virtualenv(): 2020-08-11T23:36:39.8949721Z src_prefix = os.path.join(sys.prefix, 'src') 2020-08-11T23:36:39.8951813Z else: 2020-08-11T23:36:39.8969014Z # FIXME: keep src in cwd for now (it is not a temporary folder) 2020-08-11T23:36:39.9012110Z try: 2020-08-11T23:36:39.9013489Z > src_prefix = os.path.join(os.getcwd(), 'src') 2020-08-11T23:36:39.9014538Z E FileNotFoundError: [Errno 2] No such file or directory 2020-08-11T23:36:39.9016122Z 2020-08-11T23:36:39.9017617Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/pip/_internal/locations.py:50: FileNotFoundError 2020-08-11T23:36:39.9018802Z 2020-08-11T23:36:39.9020070Z During handling of the above exception, another exception occurred: 2020-08-11T23:36:39.9020930Z 2020-08-11T23:36:39.9022275Z args = (), keywargs = {} 2020-08-11T23:36:39.9023183Z 2020-08-11T23:36:39.9024077Z @wraps(func) 2020-08-11T23:36:39.9024984Z def patched(*args, **keywargs): 2020-08-11T23:36:39.9028770Z > with self.decoration_helper(patched, 2020-08-11T23:36:39.9031861Z args, 2020-08-11T23:36:39.9038358Z keywargs) as (newargs, newkeywargs): 2020-08-11T23:36:39.9039654Z 2020-08-11T23:36:39.9040566Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/unittest/mock.py:1322: 2020-08-11T23:36:39.9041492Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Originally explored in https://github.com/simonw/datasette-notebook/issues/2#issuecomment-980789406 - I wanted an efficient way to scan a list of URLs and figure out which if any of those corresponded to Datasette tables, canned queries or SQL output that could be represented as a table on a page.

It looks like a neat way to do that is with Link: header like this:

Link: http://127.0.0.1:8058/fixtures/compound_three_primary_keys.json; rel="alternate"; type="application/datasette+json"

I can put a <link href=... in the page header too.

https://github.com/simonw/datasette/commit/6aa5886379dd9017215904fb28567b80018902f9 added the --load-extension=spatialite shortcut looking for the extension in these places:

https://github.com/simonw/datasette/blob/12877d7a48e2aa28bb5e780f929a218f7265d849/datasette/utils/init.py#L56-L60

However, in the datasetteproject/datasette docker image the file is at /usr/local/lib/mod_spatialite.so.

This results in the example command here failing:

% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=spatialite Error: Could not find SpatiaLite extension

But it does work when given an explicit path:

% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=/usr/local/lib/mod_spatialite.so INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ...

Perhaps SPATIALITE_PATHS should include /usr/local/lib/mod_spatialite.so?

I have a file named test-database (1).sqlite. When requesting the home route /, I see datasette is able to read it correctly:

However, if I click any of the links, datasette replies with: Error 404 Database not found: None

It seems the hash is crucial, as renaming the file to database (1).sqlite makes the error go away.

This lines checks for a single dash: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/views/base.py#L184

``` $ datasette test-database\ (1).sqlite INFO: Started server process [68314] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit) INFO: 127.0.0.1:54043 - "GET /favicon.ico HTTP/1.1" 200 OK INFO: 127.0.0.1:54043 - "GET / HTTP/1.1" 200 OK ... INFO: 127.0.0.1:54044 - "GET /favicon.ico HTTP/1.1" 200 OK INFO: 127.0.0.1:54044 - "GET /test-database (1) HTTP/1.1" 404 Not Found

Version: $ datasette --version datasette, version 0.53 ```

Updates the requirements on janus to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

The latest demo is now live at https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures/sortable?_facet=pk2

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1519#issuecomment-974697824

I'm going to put out 0.59.3 bugfix release with this, but I'd like to first improve the documentation on https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration to highlight the new demo.

Datasette ASGI #272 is a big part of it... but 1.0 will generally be an indicator that Datasette is a stable platform for developers to write plugins and custom templates against. So lots to think about.

For example:

• Go to https://cryptics.eigenfoo.xyz/clues/clues?_next=100 (this is the second page of results in a Datasette site)
• Search anything using the FTS search bar. For example, searching for hello will take you to https://cryptics.eigenfoo.xyz/clues/clues?_search=hello&_sort=rowid&_next=100
• A 500 Error: list index out of range is raised.

This is because the search URL includes the &_next=100 UTM parameter, carried over from where the FTS search was run. However, there isn't a second page in the search results, so a list index out of range error is raised. You can confirm that removing this UTM parameter from the URL returns the appropriate search results.

The FTS search request should strip any _next UTM parameter.

bash datasette, version 0.58.1 sqlite-utils, version 3.17

I’m currently operating under the assumption that it’s safe to allow arbitrary SQL statements because we are dealing with an immutable database. But this might not be the case - there are some pretty weird SQLite language extensions (ATTACH, PRAGMA etc) and I’m not certain they cannot be used to break things in a way that would affect future requests to the API.

Solution: provide a “safe mode” option which disables the ?sql= mechanism. This still leaves the URL filter lookups, so I need to make sure that those are “safe”.

In the future I may also implement a whitelist option where datasets can be configured to only allow specific filters against specific columns.

I'm working on a project with some HUGE (400+ lines of SQL) canned queries right now.

Any time you land on the canned query page you have to scroll down a long distance to get to the results!

Would be useful to be able to default to https://latest.datasette.io/fixtures/magic_parameters?_hide_sql=1 without needing the parameter.

I just noticed something that could make for a huge performance improvement in faceting.

The default query used by Datasette when faceting looks like this: sql select country_long, count(*) from ( select * from [global-power-plants] order by rowid ) where country_long is not null group by country_long order by count(*) desc Here it takes 53ms: https://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++country_long%2C%0D%0A++count%28%29%0D%0Afrom+%28%0D%0A++select++from+%5Bglobal-power-plants%5D+order+by+rowid%0D%0A%29%0D%0Awhere%0D%0A++country_long+is+not+null%0D%0Agroup+by%0D%0A++country_long%0D%0Aorder+by%0D%0A++count%28*%29+desc

Note that there's a order by rowid in there which isn't necessary - the order on that inner query doesn't matter since we're grouping and counting.

I had assumed SQLite would optimize this away - but it turns out it doesn't! Consider this version of the query, with that pointless order by removed: select country_long, count(*) from ( select * from [global-power-plants] ) where country_long is not null group by country_long order by count(*) desc https://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++country_long%2C%0D%0A++count%28%29%0D%0Afrom+%28%0D%0A++select++from+%5Bglobal-power-plants%5D%0D%0A%29%0D%0Awhere%0D%0A++country_long+is+not+null%0D%0Agroup+by%0D%0A++country_long%0D%0Aorder+by%0D%0A++count%28*%29+desc runs in 7.2ms!

I tried this optimization on a table with 2.5m rows in it - without the optimization it took 5 seconds, with the optimization it took 450ms. So this is a very significant improvement!

Refs #511

e.g. in https://github.com/simonw/datasette/runs/2754772233 - this is an intermittent error: ``` _ ERROR at setup of testhook_register_routes_render_message __ [gw0] linux -- Python 3.8.10 /opt/hostedtoolcache/Python/3.8.10/x64/bin/python

tmpdir = local('/tmp/pytest-of-runner/pytest-0/popen-gw0/test_hook_register_routes_rend0') request = <SubRequest 'restore_working_directory' for \<Function test_hook_register_routes_render_message>>

@pytest.fixture
def restore_working_directory(tmpdir, request):

  previous_cwd = os.getcwd()

E FileNotFoundError: [Errno 2] No such file or directory ```

Refs #1305

It looks like all I need to do to ship an alpha version to Docker Hub is NOT point the latest tag at it after it goes live: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/publish.yml#L75-L77

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1319#issuecomment-849780481

The datasette base Docker image is super convenient, but there's one problem: if any of the plugins you install require additional system dependencies (e.g., xz, git, curl) then any attempt to use apt in said Dockerfile results in an explosion:

``` $ docker-compose build Building server [+] Building 9.9s (7/9) => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 666B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 34B 0.0s => [internal] load metadata for docker.io/datasetteproject/datasette:latest 0.6s => [base 1/4] FROM docker.io/datasetteproject/datasette@sha256:2250d0fbe57b1d615a8d6df0c9d43deb9533532e00bac68854773d8ff8dcf00a 0.0s => [internal] load build context 1.8s => => transferring context: 2.44MB 1.8s => CACHED [base 2/4] WORKDIR /datasette 0.0s => ERROR [base 3/4] RUN apt-get update && apt-get install --no-install-recommends -y git ssh curl xz-utils 9.2s

[base 3/4] RUN apt-get update && apt-get install --no-install-recommends -y git ssh curl xz-utils:

6 0.446 Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]

6 0.449 Get:2 http://deb.debian.org/debian buster InRelease [121 kB]

6 0.459 Get:3 http://httpredir.debian.org/debian sid InRelease [157 kB]

6 0.784 Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]

6 0.790 Get:5 http://httpredir.debian.org/debian sid/main amd64 Packages [8626 kB]

6 1.003 Get:6 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]

6 1.180 Get:7 http://security.debian.org/debian-security buster/updates/main amd64 Packages [286 kB]

6 7.095 Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [10.9 kB]

6 8.058 Fetched 17.2 MB in 8s (2243 kB/s)

6 8.058 Reading package lists...

6 9.166 E: flAbsPath on /var/lib/dpkg/status failed - realpath (2: No such file or directory)

6 9.166 E: Could not open file - open (2: No such file or directory)

6 9.166 E: Problem opening

6 9.166 E: The package lists or status file could not be parsed or opened.

```

The problem seems to be from completely wiping out /var/lib/dpkg in the upstream Dockerfile:

https://github.com/simonw/datasette/blob/1b697539f5b53cec3fe13c0f4ada13ba655c88c7/Dockerfile#L18

I've tested without removing the directory and apt works as expected.

https://nhs-england-hospitals.now.sh

and an associated map visualisation:

http://run.plnkr.co/preview/cj9zlf1qc0003414y90ajkwpk/

Datasette is wonderful!

Working on my PR i noticed that tests are very slow.

The plain pytest run took about 37 minutes for me. However i could shave of about 10 minutes from that if i used pytest-xdist to parallelize execution. pytest -n 8 is run only in 28 minutes on my machine.

I can create a PR to mention that in your documentation. This will be a simple change to add pytest-xdist to requirements and change a command to run pytest in documentation.

Does that make sense to you?

After a bit more investigation it looks like python-xdist is not an answer. It creates a race condition for tests that try to clead temp dir before run.

Profiling shows that most time is spent on conn.executescript(TABLES) in make_app_client function. Which makes sense.

Perhaps the better approach would be look at the app_client fixture which is already session scoped, but not used by all test cases. And/or use conn = sqlite3.connect(":memory:") which is much faster. And/or truncate tables after each TC instead of deleting the file and re-creating them.

I can take a look which is the best approach if you give the go-ahead.

Currently throws an ugly error message: (datasette-graphql) datasette-graphql % datasette fivethirtyeight.db -p 80094 INFO: Started server process [45497] INFO: Waiting for application startup. INFO: Application startup complete. Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-graphql-n1OSJCS8/bin/datasette", line 8, in <module> sys.exit(cli()) ... server = await loop.create_server( File "/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py", line 1461, in create_server sock.bind(sa) OverflowError: bind(): port must be 0-65535.

Hi, If I run

docker run -p 8001:8001 -v `pwd`:/mnt \ 1 ↵ datasetteproject/datasette \ datasette -p 8001 -h 0.0.0.0 fixtures.db

I have

Error: Invalid value for '[FILES]...': Path 'fixtures.db' does not exist.

If I run test -f fixtures.db && echo "it exists." I have it exists..

What's my error?

Thank you

• compact dict and set building
• remove redundant parentheses
• simplify chained conditions
• change method name to lowercase
• use triple double quotes for docstrings

please feel free to accept/reject any of these independent commits

With the current configuration Prettier seems to be installed on every run - which can been seen from the output:

npx: installed 1 in 5.166s

Prettier isn't explicitly being installed (it's surprising that actually installing the dependencies isn't included in the actions/cache docs) but it turns out that npx will automatically install the package for the specified command (it actually guesses the package name from the name of the command). I'm not sure where Prettier ends up being installed but it doesn't appear to be in ~/.npm according to the post-cache output (or ./node_modules when I tested locally):

Cache hit occurred on the primary key Linux-npm-565329898f77080e58b14d45cf816ab94877e6f2ece9d395c369c533548a7ee7, not saving cache.

I think there are a couple of approaches to tackling this, you could manually install/cache Prettier within the action, or add a package.json with Prettier. I would go with the latter because it's a more standard and maintainable approach and it will also ensure that, along with CI, anyone working on the project will run the same version of Prettier (you'll also get Dependabot JavaScript updates).

I've tested the package.json approach on a branch and am happy to turn it into a pull request if you fancy.

e.g. I should be able to do this:

datasette App/data.db Other_App/data.db

This currently errors because you can't have two databases taking the /data URL path.

Instead, how about in this particular case assigning the second database /data-1?

Hi Simon! Maybe it's just me, but when using _searchmode=raw (trying to enable wildcard-searching) in combination with the "_search_COLUMN"-table argument, I get a list index out of range error. When combining with the simpler "_search"-argument everything works, including wildcard-seaches.. Here's the traceback:

``` Traceback (most recent call last): File "/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/utils/asgi.py", line 122, in route_path return await view(new_scope, receive, send) File "/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/utils/asgi.py", line 196, in view request, scope["url_route"]["kwargs"] File "/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/base.py", line 204, in get request, database, hash, correct_hash_provided, kwargs File "/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/base.py", line 342, in view_get request, database, hash, **kwargs File "/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/table.py", line 393, in data search_col = key.split("search", 1)[1] IndexError: list index out of range

```

https://cloud.google.com/run/quotas lists the maximum response size as 32MB.

I spotted a bug where attempting to download a database file larger than that from a Cloud Run deployment (in this case it was https://github-to-sqlite.dogsheep.net/github.db after I accidentally increased the size of that database) returned a 500 error because of this.

I'd like to redirect https://docs.datasette.io/en/stable/config.html to a new https://docs.datasette.io/en/stable/settings.html page too. I can use https://docs.readthedocs.io/en/stable/user-defined-redirects.html for that.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1105#issuecomment-733190827

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1033#issuecomment-714681365

Refs #1023

• [x] Remove the /-/...blob/... route I added in #1040 in place of the new .blob renderer URLs
• [x] Link to new .blob download links on the arbitrary query page (using _blob_hash=...) - plus tests for this

Closes #1050, Closes #1051

To handle packages that require Javascript state setting prior to loading a package (eg thebelab, provide a template block before the URLs are loaded.

Hi, thanks for datasette!

This PR adds the LICENSE to source distributions, which seems the norm for Apache-2.0 stuff.

I noticed the 0.50.2 sdist doesn't ship LICENSE, but the 0.5.2 whl does, so I'm assuming the intent is to ship... and it's a one-liner!

Motivation:

It might be a bit of a slog, but I'm looking to see about getting datasette (and friends!) available on conda-forge. There are a few missing upstreams (asgi-csrf, python-basecov, mergedeep) and some of the plugins don't even appear to have tarballs (just whl!), but the little stuff like licenses are nice to get out handled upstream vs separately grabbing them.

I just upgraded from 0.49 to 0.50.1 and found that the links on column headers are broken.

If I inspect the source, they have a leading "//" (without host or port) rather than including base_url like other links on the page do. The links in the "gears" menu for each column do work.

I don't have custom templates for my project.

This is a request for a "convenience" feature, and only a nice to have. It's based on seeing this feature in several little command line hypertext server apps.

If you run, for example:

datasette.exe serve --open "mydb.s3db"

I would like it if default browser is launched, at the URL that is being served.

The angular cli does this, for example

ng serve <project> --open #see https://angular.io/cli/serve

...as does my usual mini web server of choice when inspecting local static files....

npx http-server -o # see https://www.npmjs.com/package/http-server

Just a tiny thing. Love your work!

Steps to reproduce: visit https://2a4b892.datasette.io/fixtures/roadside_attractions?_facet_m2m=attraction_characteristic and click "Apply"

Result is a 500: no such column: attraction_characteristic

The error occurs because of this hidden HTML input:

<input type="hidden" name="_facet" value="attraction_characteristic">

This should be:

<input type="hidden" name="_facet_m2m" value="attraction_characteristic">

The HTML form for a read-only canned query includes the hidden CSRF token field added in #798 for writable canned queries (#698).

This means that submitting those read-only forms exposes the CSRF token in the URL - for example on https://latest.datasette.io/fixtures/neighborhood_search submitting the form took me to:

https://latest.datasette.io/fixtures/neighborhood_search?text=down&csrftoken=IlFubnoxVVpLU1NGT3NMVUoi.HbOPd2YH_epQmp8f_aAt0s-MxtU

This token could potentially leak to an attacker if the resulting page has a link to an external site on it and the user clicks the link, since the token would be exposed in the referral logs.

Is there any reason NOT to apply white-space: pre-wrap to the contents of all table cells in Datasette?

The default display mechanism of HTML (stripping leading/trailing slashes and collapsing all other whitespace) doesn't really make sense for displaying the kind of data that Datasette works with.

Need a cookie clearing mechanism and a way to show that you are logged in.

datasette-auth-github had a solution for this that can be pulled into core.

The .coverage file generated by running pytest-cov is now a SQLite database!

I could do something interesting with this. Maybe after each test run for a new commit I could store that database file somewhere?

Lots of interesting challenges here.

I got a change into coveragepy last year which helps make the custom SQL functions available for doing fun things in Datasette: https://github.com/nedbat/coveragepy/issues/868

Bigger challenge: if I have a DB file for every commit, that's hundreds (potentially thousands) of DB files. Datasette isn't designed to handle thousands of files like that.

So, do I figure out how to have Datasette open a file on-command for just a single request? Or, an easier option, do I copy data from those files into a single database with a modified schema to include the commit hash in each table row?

(Following on from #841 and #844)

https://github.com/simonw/datasette-auth-github/issues/62 will work for this.

The only URLs that should be available without authentication at all times are the /-/static/ prefix, to allow for HTTP caching.

Refs #787. Will need quite a bit of manual testing since this involves code which runs against Heroku and Cloud Run.

e.g. https://latest.datasette.io/fixtures/compound_three_primary_keys.tsv?_size=max

The HTML starts like this:

```html

internals.rst will be landing as part of #683

I would like is the ability for renderers to opt-in / opt-out of being displayed as options on the page.

https://www.niche-museums.com/browse/museums for example shows a atom link because the datasette-atom plugin is installed... but clicking it will give you a 400 error because the correct columns are not present.

Here's the code that passes a list of renderers to the template:

https://github.com/simonw/datasette/blob/2d099ad9c657d2cab59de91cdb8bfed2da236ef6/datasette/views/base.py#L411-L423

A renderer is currently defined as a two-key dictionary: python @hookimpl def register_output_renderer(datasette): return { 'extension': 'test', 'callback': render_test } I can add a third key, "should_suggest" which is a function that returns True or False for a given query. If that key is missing it is assumed to return True.

One catch: what arguments should be passed to the should_suggest(...) function?

UPDATE: now calling it can_render instead.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/581#issuecomment-634856748

I have some tables where I'd like the default page size to be 10, without affecting the rest of my Datasette instance.

datasette-atom only works if the user constructs a SQL query with specific output columns (atom_id ,atom_updated etc).

It would be good if the .atom link wasn't shown on the query/table page unless those columns were present. Right now you get a link which results in a 400 error:

See also #581.

Thank you very much for this project.

I have created some canned queries but some of the filters include a colon eg. "com.ubuntu.cloud:server:18.04:amd64". When saved these colons are parsed as named parameters.

Is there a way to escape colons in a canned query?

Refs #739 - metadata.yml or metadata.yaml should be detected in the same way as metadata.json is.

Spotted in #732. This should not return any results... but it does:

https://latest.datasette.io/fixtures/searchable?_search=bar%2A&_trace=1

The query from trace is: "sql": "select count(*) from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search))", "params": { "search": "bar*" }

Refs #648. TODO: - [x] Pass a view_name to render_template() - [x] Mechanism for custom status code / headers / redirect - [x] Documentation

Would allow people who want to host private data to do so. .sh

Hi Simon. I need some help with both understanding and adding http-headers. If I call datasette on localhost with --config default_cache_ttl:120 and --cors, I only get the following response-headers:

access-control-allow-origin: * content-type: text/html; charset=utf-8 date: Sat, 22 Feb 2020 10:32:15 GMT referrer-policy: no-referrer server: uvicorn transfer-encoding: chunked

Cors works, but no caching-header is set? Same thing happens if I use the command in a Dockerfile and run datasette with docker.

Second, how can one add headers to uvicorn? I've tried to add uvicorn commands to the Dockerfile, before the final datasette command, but it doesn't work. Is there any way to add headers to the uvicorn.run() command i datasette? I particular, I would like to add some of the missing security-headers:

Thank you for a great product!

To allow plugins to customize how values matching a specific pattern are displayed in the HTML table view.

See #595, #594, #404.

The big thing holding me back from ditching Python 3.5 was glitch.com - but they now offer Python 3.7: https://support.glitch.com/t/can-you-upgrade-python-to-latest-version/7980/25?u=simonw

Hi, I defined some custom queries in my metadata.json. There are Chinese characters in the names of the queries. So the urls are like http://127.0.0.1:8001/mydb/测试查询. When opening such urls, datasette will throw an exception. Traceback (most recent call last): File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/utils/asgi.py", line 100, in __call__ return await view(new_scope, receive, send) File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/utils/asgi.py", line 172, in view request, **scope["url_route"]["kwargs"] File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/views/base.py", line 267, in get request, database, hash, correct_hash_provided, **kwargs File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/views/base.py", line 471, in view_get for key in self.ds.renderers.keys() File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/views/base.py", line 471, in <dictcomp> for key in self.ds.renderers.keys() File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/utils/__init__.py", line 655, in path_with_format path = request.path File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/utils/asgi.py", line 49, in path self.scope.get("raw_path", self.scope["path"].encode("latin-1")) UnicodeEncodeError: 'latin-1' codec can't encode characters in position 9-11: ordinal not in range(256) This used to work when datasette was based on sanic.

Btw, thanks for the great work!

While debugging why my static mounts using a relative path (--static mystatic:rel/path/to/dir) not working, I noticed that the requests fail no matter what, returning 404 errors.

The reason is that datasette tries to prevent traversal exploits by checking if the path is relative to its registered directory. This check fails when the mount is a relative directory, because /abs/dir/file obviously not under dir/file.

https://github.com/simonw/datasette/blob/81fa8b6cdc5457b42a224779e5291952314e8d20/datasette/utils/asgi.py#L303-L306

This also has the consequence of returning any requested file, because when /abs/dir/../../evil.file resolves aiofiles happily returns it to the client after it resolves the path itself. The solution is to make sure we're checking relativity of paths after they're fully resolved.

I've implemented the mentioned changes and also updated the tests.

It's not clear to me what this endpoint should do now as a result of #419 - it's still useful to be able to introspect databases for tools like datasette-registry, but since we aren't pre-calculating introspection data any more I need to rethink the approach.

For one thing, this endpoint may need to be paginated. Or maybe it should be split up into separate endpoints for each connected database? Those should probably be paginated too seeing as fivethirtyeight has 400+ tables.

ujson is already a dependency of Sanic, and should be quite a bit faster.

Parent: #354

https://github.com/simonw/datasette/blob/56623e48da5412b25fb39cc26b9c743b684dd968/datasette/app.py#L517-L519 throws an error if it gets an empty list back. Simplest solution is to write a helper func that just says

python result = list(await self.execute(name, sql, params) if result: return result[0][0]

and use it anywhere [0][0] is now.

This is the last part of #419 - with the move to supporting mutable databases by default, the inspect-data mechanism currently in use no-longer makes much sense.

The one optimization I think it's worth keeping for databases opened in immutable mode is the cached table counts. I think datasette inspect should cut down to only counting the rows in the tables - the other things done by inspect (figuring out columns, foreign key relationships, FTS etc) should all be fast enough that they can be reliably performed at runtime even against large databases.

If performing them at run-time has performance issues, I would rather cache those results internally within Datasette after they are first calculated than continue to support them in the datasette inspect command - to keep things simpler.

This is a bit tricky, because unlike Now there doesn't seem to be a way to tell Hyper to "build this Dockerfile and deploy the resulting image". They expect you to build a container and publish it to a registry instead.

https://docs.hyper.sh/Reference/CLI/load.html allows you to publish an image directly from a tarball, but that still leaves the challenge of creating that image. The nice thing about the Now integration is that you don't need to have Docker installed on your local machine.

My datasette-cluster-map plugin currently works by detecting latitude and longitude columns. I'd like to be able to configure it to look for different column names.

One way to do this could be to support optional plugin configuration as part of metadata.json. Something like this:

{
    "title": "Polar Bear Ear Tags, 2009-2011",
    "source": "USGS Alaska Science Center, Polar Bear Research Program",
    "source_url": "https://alaska.usgs.gov/products/data.php?dataid=130",
    "plugins": {
        "datasette_cluster_map": {
            "latitude_columns": [
                "latitude",
                "Capture Latitude"
            ],
            "longitude_columns": [
                "longitude",
                "Capture Longitude"
            ]
        }
    }
}

These settings should be supported at the root level or at the individual database or table level.

They could also be exposed in the https://datasette-cluster-map-demo.now.sh/-/plugins debug tool.

Refs #14

Features like faceting, foreign key expansions and now the inspect-less index view mean Datasette can end up executing a surprisingly large number of SQL queries to render a single page.

Past experience with projects like tikbar have shown that being able to see what actually went into rendering a page can be critical for optimizing performance and generally understanding how everything works.

Support a tracing mode (probably via a ?_trace=1 querystring) which adds information about what is actually going on to both the HTML and the JSON.

WIP for #427

In moving away from the existing static inspect method (see #420 and #419) the biggest thing lost is full table row counts. These can be expensive against large tables, but currently Datasette runs the count (*) from x query once at inspection time and then reuses it for every page.

We can run those counts with a timelimit, but this means that for larger tables we won't be able to show a count at all, which is disappointing.

Is there a way we can find an approximate or lower bound count for a table?

It would be useful if Datasette could spot when a SQLite database file changes on disk and restart itself (hence re-running .inspect() and picking up the new content hash). Ideally this could happen in an atomic way so no requests get dropped during the switch-over.

This may not play well with SQLite opening databases in immutable mode. Research required.

It would be great if the metadata file allowed you to enter a description for the query. We have a lot of pre-defined queries that can only be so descriptive by their name. It would be nice if an optional description could be included underneath the name within the UI, or on hover where it currently shows the SQL.

The default preview of a database shows all columns (is the row count limited?) which is fine in many cases but can take a long time to load / offer a large overhead if the table is a SpatiaLite table containing geometry columns that include large shapefiles.

Would it make sense to have a setting that can limit the amount of text displayed in any given cell in the table preview, or (less useful?) suppress (with notification) the display of overlong columns unless enabled by the user?

An issue then arises if a user does want to see all the text in a cell:

1) for a particular cell; 2) for every cell in the table; 3) for all cells in a particular column or columns

(I haven't checked but what if a column contains e.g. raw image data? Does this display as raw data? Or can this be rendered in a context aware way as an image preview? I guess a custom template would be one way to do that?)

e.g. on https://fivethirtyeight.datasettes.com/fivethirtyeight-ac35616/nba-elo%2Fnbaallelo.json?_facet=lg_id&_size=0 { "facet_results": { "lg_id": { "name": "lg_id", "results": [ { "value": "NBA", "label": "NBA", "count": 118016, "toggle_url": "http://fivethirtyeight.datasettes.com/fivethirtyeight-ac35616/nba-elo%2Fnbaallelo.json?_facet=lg_id&_size=1&lg_id=NBA", "selected": false }, { "value": "ABA", "label": "ABA", "count": 8298, "toggle_url": "http://fivethirtyeight.datasettes.com/fivethirtyeight-ac35616/nba-elo%2Fnbaallelo.json?_facet=lg_id&_size=1&lg_id=ABA", "selected": false } ], "truncated": false } }, "suggested_facets": [ { "name": "_iscopy", "toggle_url": "/fivethirtyeight-ac35616/nba-elo%2Fnbaallelo.json?_facet=lg_id&_size=1&_facet=_iscopy" } ], "next_url": "http://fivethirtyeight.datasettes.com/fivethirtyeight-ac35616/nba-elo%2Fnbaallelo.json?_facet=lg_id&_size=1&_next=1", } next_url and facet_results both link to http:// when they should link to https://.

Note that suggested facets doesn't include the full URL at all, which is a consistency bug.

This is needed for #266 - if a table name ends with .json or .csv right now our URL pattern matching will do the wrong thing.

We should be smarter about this. This does mean we will have some URLs that look like this:

http://localhost:8001/dbname/weird.json - returning HTML, not JSON
http://localhost:8001/dbname/weird.json.json - returning JSON
http://localhost:8001/dbname/weird.json.csv - returning CSV

Causes the specified columns in the output to be treated as JSON, and returned deserialized in the .json or .jsono response.

This will be particularly powerful when combined with https://sqlite.org/json1.html

I think that's all for getting Versioneer support, I've been happily using it in a couple of projects ...

In [2]: datasette.__version__ Out[2]: '0.22+3.g6e12445' Repo: https://github.com/warner/python-versioneer

Versioneer Licence: Public Domain (CC0-1.0)

Closes #273