Codecov Report

Merging #1617 (af293c9) into main (2aa686c) will increase coverage by 0.06%. The diff coverage is 100.00%.

```diff @@ Coverage Diff @@

main #1617 +/-

========================================== + Coverage 92.09% 92.16% +0.06%
========================================== Files 34 34
Lines 4518 4531 +13
========================================== + Hits 4161 4176 +15
+ Misses 357 355 -2
```

| Impacted Files | Coverage Δ | | |---|---|---| | datasette/app.py | 95.37% <100.00%> (ø) | | | datasette/views/table.py | 96.19% <0.00%> (ø) | | | datasette/utils/__init__.py | 94.79% <0.00%> (+<0.01%) | :arrow_up: | | datasette/views/base.py | 95.49% <0.00%> (+0.07%) | :arrow_up: | | datasette/views/special.py | 95.09% <0.00%> (+2.38%) | :arrow_up: |

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2aa686c...af293c9. Read the comment docs.

I've decided not to do this, because of the risk that Cloudflare could cache the JSON version for an HTML page or vice-versa.

Codecov Report

Merging #1626 (4b4d0e1) into main (b5e6b1a) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #1626 +/-

======================================= Coverage 92.16% 92.16%
======================================= Files 34 34
Lines 4531 4531
======================================= Hits 4176 4176
Misses 355 355
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update b5e6b1a...4b4d0e1. Read the comment docs.

That broke on a macOS test: https://github.com/simonw/datasette/runs/5044036993?check_suite_focus=true

I'm going to remove macOS and Ubuntu and just try Windows purely to see what happens there.

First stable Black release!

Here's the new test: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/tests/test_html.py#L927-L936

It looks like this is because IndexView extends BaseView rather than extending DataView which is where all that CORS stuff happens:

https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/datasette/views/index.py#L18-L21

Another thing I should address with the refactor project in: - #878

I ran the following on https://www.google.com/ in the console to demonstrate that these work as intended:

javascript [ "https://latest.datasette.io/fixtures", "https://latest.datasette.io/fixtures?sql=select+1", "https://latest.datasette.io/fixtures/facetable" ].forEach(async (url) => { response = await fetch(url, {method: "HEAD"}); console.log(response.headers.get("Link")); });

A few other pages do that too, including: - https://latest.datasette.io/-/messages - https://latest.datasette.io/-/allow-debug

Codecov Report

Merging #1622 (fbaf317) into main (8d5779a) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #1622 +/-

======================================= Coverage 92.11% 92.11%
======================================= Files 34 34
Lines 4525 4525
======================================= Hits 4168 4168
Misses 357 357
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8d5779a...fbaf317. Read the comment docs.

% curl -s -I 'https://latest.datasette.io/' | grep link link: https://latest.datasette.io/.json; rel="alternate"; type="application/json+datasette" % curl -s -I 'https://latest.datasette.io/fixtures' | grep link link: https://latest.datasette.io/fixtures.json; rel="alternate"; type="application/json+datasette" % curl -s -I 'https://latest.datasette.io/fixtures?sql=select+1' | grep link link: https://latest.datasette.io/fixtures.json?sql=select+1; rel="alternate"; type="application/json+datasette" % curl -s -I 'https://latest.datasette.io/-/plugins' | grep link link: https://latest.datasette.io/-/plugins.json; rel="alternate"; type="application/json+datasette"

I just realized I can refactor this to make it much simpler.

Thanks for looking into this. It might have been nice if explain surfaced these function calls. Looks like explain query plan does, but only for basic queries.

``` sqlite-utils fixtures.db 'explain query plan select * from pragma_function_list(), pragma_database_list(), pragma_module_list()' -t id parent notused detail

4 0 0 SCAN pragma_function_list VIRTUAL TABLE INDEX 0: 8 0 0 SCAN pragma_database_list VIRTUAL TABLE INDEX 0: 12 0 0 SCAN pragma_module_list VIRTUAL TABLE INDEX 0: ```

``` sqlite-utils fixtures.db 'explain query plan select * from pragma_function_list() as fl, pragma_database_list() as dl, pragma_module_list() as ml' -t id parent notused detail

4 0 0 SCAN fl VIRTUAL TABLE INDEX 0: 8 0 0 SCAN dl VIRTUAL TABLE INDEX 0: 12 0 0 SCAN ml VIRTUAL TABLE INDEX 0: ```

Demos - these pages both have <link rel=... if you view source on them:

• https://latest.datasette.io/fixtures/sortable
• https://latest.datasette.io/fixtures/sortable/a,a

And you can hit them with curl like so: ``` % curl -I 'https://latest.datasette.io/fixtures/sortable'
HTTP/1.1 200 OK link: https://latest.datasette.io/fixtures/sortable.json; rel="alternate"; type="application/json+datasette" cache-control: max-age=5 referrer-policy: no-referrer access-control-allow-origin: * access-control-allow-headers: Authorization access-control-expose-headers: Link content-type: text/html; charset=utf-8 x-databases: _memory, _internal, fixtures, extra_database Date: Wed, 02 Feb 2022 07:56:17 GMT Server: Google Frontend Transfer-Encoding: chunked

% curl -I 'https://latest.datasette.io/fixtures/sortable/a,a' HTTP/1.1 200 OK link: https://latest.datasette.io/fixtures/sortable/a,a.json; rel="alternate"; type="application/json+datasette" cache-control: max-age=5 referrer-policy: no-referrer access-control-allow-origin: * access-control-allow-headers: Authorization access-control-expose-headers: Link content-type: text/html; charset=utf-8 x-databases: _memory, _internal, fixtures, extra_database Date: Wed, 02 Feb 2022 07:56:24 GMT Server: Google Frontend Transfer-Encoding: chunked ```

Documentation: https://docs.datasette.io/en/latest/json_api.html#discovering-the-json-for-a-page

https://docs.datasette.io/en/latest/json_api.html top --cors section mentions the new Access-Control-Expose-Headers: Link header.

I've been thinking about this a bunch too. If I build anything along these lines it will be as part of the Datasette Cloud hosted service I'm working on, maybe as a free tier.

I was hoping that explain select ... might be able to easily spot when people are calling PRAGMA functions, but this output doesn't look very helpful: ``` % sqlite-utils fixtures.db 'explain select * from pragma_database_list()' -t addr opcode p1 p2 p3 p4 p5 comment

 0  Init            0    11     0                        0
 1  VOpen           0     0     0  vtab:7F9C90AC3070     0
 2  Integer         0     1     0                        0
 3  Integer         0     2     0                        0
 4  VFilter         0    10     1                        0
 5  VColumn         0     0     3                        0
 6  VColumn         0     1     4                        0
 7  VColumn         0     2     5                        0
 8  ResultRow       3     3     0                        0
 9  VNext           0     5     0                        0
10  Halt            0     0     0                        0
11  Transaction     0     0    35  0                     1
12  Goto            0     1     0                        0

```

I also need to consider if supposedly harmless side-effect free pragma functions could be used to work around the Datasette permissions system. My hunch is that wouldn't be a problem, because if you're allowing arbitrary SQL queries you're already letting people ignore the permissions system.

One example: ``` sqlite-utils fixtures.db 'pragma database_list' -t seq name file

0  main    /Users/simon/Dropbox/Development/datasette/fixtures.db

``` Though it looks like I already allow-listed that one in #761: https://latest.datasette.io/_memory?sql=select+*+from+pragma_database_list%28%29

I also just realized that pragma pragma_list can be used to generate a list of all known pragmas for the connection:

sqlite-utils fixtures.db 'pragma pragma_list' --fmt github

| name | |---------------------------| | analysis_limit | | application_id | | auto_vacuum | | automatic_index | | busy_timeout | | cache_size | | cache_spill | | case_sensitive_like | | cell_size_check | | checkpoint_fullfsync | | collation_list | | compile_options | | count_changes | | data_version | | database_list | | default_cache_size | | defer_foreign_keys | | empty_result_callbacks | | encoding | | foreign_key_check | | foreign_key_list | | foreign_keys | | freelist_count | | full_column_names | | fullfsync | | function_list | | hard_heap_limit | | ignore_check_constraints | | incremental_vacuum | | index_info | | index_list | | index_xinfo | | integrity_check | | journal_mode | | journal_size_limit | | legacy_alter_table | | lock_proxy_file | | locking_mode | | max_page_count | | mmap_size | | module_list | | optimize | | page_count | | page_size | | pragma_list | | query_only | | quick_check | | read_uncommitted | | recursive_triggers | | reverse_unordered_selects | | schema_version | | secure_delete | | short_column_names | | shrink_memory | | soft_heap_limit | | synchronous | | table_info | | table_list | | table_xinfo | | temp_store | | temp_store_directory | | threads | | trusted_schema | | user_version | | wal_autocheckpoint | | wal_checkpoint | | writable_schema |

So I could use that list to create a much more specific regular expression, which would then allow the word "pragma" to be used more freely while still protecting against any known pragma function being called.

But... I just noticed something I had missed in the docs for https://www.sqlite.org/pragma.html#pragfunc

Table-valued functions exist only for PRAGMAs that return results and that have no side-effects.

So it's possible I'm being overly paranoid here after all: what I want to block here is people running things like PRAGMA case_sensitive_like = 1 which could affect the global state for that connection and cause unexpected behaviour later on.

So maybe I should allow all pragma functions. I previously allowed an allow-list of them in: - #761

There's a workaround for this at the moment, which is to use parameterized SQL queries. For example, this:

https://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+%3D+%3Atitle&title=The+Pragmatic+Programmer

So the SQL query is select * from books where title = :title and then &title=... is added to the URL.

The reason behind the quite aggressive pragma filtering is that SQLite allows you to execute pragmas using function calls, like this one:

sql SELECT * FROM pragma_index_info('idx52'); These can be nested arbitrarily deeply in sub-queries, so it's difficult to write a regular expression that will definitely catch them.

I'm open to relaxing the regex a bit, but I need to be very confident that it's safe to do so.

Running it as part of datasette publish is a smart idea - I'm slightly nervous about modifying the database file that has been published though, since part of the undocumented contract right now is that the bytes served are the exact same bytes as the ones you ran the publish against.

But there's no reason for that expectation to exist, and I doubt anyone is relying on that.

Weirdly the bug does NOT exhibit itself on this demo: https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1 - which correctly links to https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1.json

It's not just the table with slashes in the name. Same thing on http://127.0.0.1:3344/foo/bar/fixtures/attraction_characteristic/1 - the json link goes to a JSON-rendered 404 on http://127.0.0.1:3344/foo/bar/foo/bar/fixtures/attraction_characteristic/1.json

Here's what I was hacking around with when I uncovered this problem: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 77fb285..8c57d08 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import urllib import itertools import json @@ -615,44 +616,37 @@ class TableView(RowTableShared): if request.args.get("_timelimit"): extra_args["custom_time_limit"] = int(request.args.get("_timelimit"))

• Execute the main query!

• results = await db.execute(sql, params, truncate=True, **extra_args)

• Calculate the total count for this query

• filtered_table_rows_count = None
• if (
• not db.is_mutable
• and self.ds.inspect_data
• and count_sql == f"select count(*) from {table} "
• ):

• We can use a previously cached table row count

• try:
• filtered_table_rows_count = self.ds.inspect_data[database]["tables"][
• table
• ]["count"]
• except KeyError:

• pass

• Otherwise run a select count(*) ...

• if count_sql and filtered_table_rows_count is None and not nocount:
• try:
• count_rows = list(await db.execute(count_sql, from_sql_params))
• filtered_table_rows_count = count_rows[0][0]
• except QueryInterrupted:

• pass

• Faceting

• if not self.ds.setting("allow_facet") and any(
• arg.startswith("_facet") for arg in request.args
• ):
• raise BadRequest("_facet= is not allowed")
• async def execute_count():

• Calculate the total count for this query

• filtered_table_rows_count = None
• if (
• not db.is_mutable
• and self.ds.inspect_data
• and count_sql == f"select count(*) from {table} "
• ):

• We can use a previously cached table row count

• try:
• filtered_table_rows_count = self.ds.inspect_data[database][
• "tables"
• ][table]["count"]
• except KeyError:
• pass +
• if count_sql and filtered_table_rows_count is None and not nocount:
• try:
• count_rows = list(await db.execute(count_sql, from_sql_params))
• filtered_table_rows_count = count_rows[0][0]
• except QueryInterrupted:
• pass +
• return filtered_table_rows_count +

• filtered_table_rows_count = await execute_count()

   # pylint: disable=no-member
   facet_classes = list(
       itertools.chain.from_iterable(pm.hook.register_facet_classes())
   )
  

  • facet_results = {}
  • facets_timed_out = [] facet_instances = [] for klass in facet_classes: facet_instances.append( @@ -668,33 +662,58 @@ class TableView(RowTableShared): ) )

• if not nofacet:

• for facet in facet_instances:
• (
• instance_facet_results,
• instance_facets_timed_out,
• ) = await facet.facet_results()
• for facet_info in instance_facet_results:
• base_key = facet_info["name"]
• key = base_key
• i = 1
• while key in facet_results:
• i += 1
• key = f"{base_key}_{i}"
• facet_results[key] = facet_info

• facets_timed_out.extend(instance_facets_timed_out)

• Calculate suggested facets

• suggested_facets = []
• if (
• self.ds.setting("suggest_facets")
• and self.ds.setting("allow_facet")
• and not _next
• and not nofacet
• and not nosuggest
• ):
• for facet in facet_instances:
• suggested_facets.extend(await facet.suggest())
• async def execute_suggested_facets():

• Calculate suggested facets

• suggested_facets = []
• if (
• self.ds.setting("suggest_facets")
• and self.ds.setting("allow_facet")
• and not _next
• and not nofacet
• and not nosuggest
• ):
• for facet in facet_instances:
• suggested_facets.extend(await facet.suggest())
• return suggested_facets +
• async def execute_facets():
• facet_results = {}
• facets_timed_out = []
• if not self.ds.setting("allow_facet") and any(
• arg.startswith("_facet") for arg in request.args
• ):
• raise BadRequest("_facet= is not allowed") +
• if not nofacet:
• for facet in facet_instances:
• (
• instance_facet_results,
• instance_facets_timed_out,
• ) = await facet.facet_results()
• for facet_info in instance_facet_results:
• base_key = facet_info["name"]
• key = base_key
• i = 1
• while key in facet_results:
• i += 1
• key = f"{base_key}_{i}"
• facet_results[key] = facet_info
• facets_timed_out.extend(instance_facets_timed_out) +
• return facet_results, facets_timed_out +

• Execute the main query, facets and facet suggestions in parallel:

• (
• results,
• suggested_facets,
• (facet_results, facets_timed_out),
• ) = await asyncio.gather(
• db.execute(sql, params, truncate=True, **extra_args),
• execute_suggested_facets(),
• execute_facets(),
• ) +
• results = await db.execute(sql, params, truncate=True, **extra_args)

   # Figure out columns and rows for the query
   columns = [r[0] for r in results.description]
  

  ``` It's a hacky attempt at running some of the table page queries in parallel to see what happens.

Prototype, not sure that this actually works yet: diff diff --git a/datasette/database.py b/datasette/database.py index 6ce8721..0c4aec7 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -256,18 +256,26 @@ class Database: # Try to get counts for each table, $limit timeout for each count counts = {} for table in await self.table_names(): - try: - table_count = ( - await self.execute( - f"select count(*) from [{table}]", - custom_time_limit=limit, - ) - ).rows[0][0] - counts[table] = table_count - # In some cases I saw "SQL Logic Error" here in addition to - # QueryInterrupted - so we catch that too: - except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError): - counts[table] = None + print(table.lower()) + if table.lower() == "knn": + counts[table] = 0 + else: + try: + table_count = ( + await self.execute( + f"select count(*) from [{table}]", + custom_time_limit=limit, + ) + ).rows[0][0] + counts[table] = table_count + # In some cases I saw "SQL Logic Error" here in addition to + # QueryInterrupted - so we catch that too: + except ( + QueryInterrupted, + sqlite3.OperationalError, + sqlite3.DatabaseError, + ): + counts[table] = None if not self.is_mutable: self._cached_table_counts = counts return counts

Prototype: diff diff --git a/datasette/app.py b/datasette/app.py index 09d7d03..e2a5aea 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -724,6 +724,47 @@ class Datasette: sqlite_extensions[extension] = None except Exception: pass + # More details on SpatiaLite + if "spatialite" in sqlite_extensions: + spatialite_details = {} + fns = ( + "spatialite_version", + "spatialite_target_cpu", + "rcheck_strict_sql_quoting", + "freexl_version", + "proj_version", + "geos_version", + "rttopo_version", + "libxml2_version", + "HasIconv", + "HasMathSQL", + "HasGeoCallbacks", + "HasProj", + "HasProj6", + "HasGeos", + "HasGeosAdvanced", + "HasGeosTrunk", + "HasGeosReentrant", + "HasGeosOnlyReentrant", + "HasMiniZip", + "HasRtTopo", + "HasLibXML2", + "HasEpsg", + "HasFreeXL", + "HasGeoPackage", + "HasGCP", + "HasTopology", + "HasKNN", + "HasRouting", + ) + for fn in fns: + try: + result = conn.execute("select {}()".format(fn)) + spatialite_details[fn] = result.fetchone()[0] + except Exception: + pass + sqlite_extensions["spatialite"] = spatialite_details + # Figure out supported FTS versions

I'm going to apply the hack, then fix it again in: - #1518