html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/1534#issuecomment-1028461220,https://api.github.com/repos/simonw/datasette/issues/1534,1028461220,IC_kwDOBm6k_c49TRKk,9599,2022-02-02T23:39:33Z,2022-02-02T23:39:33Z,OWNER,"I've decided not to do this, because of the risk that Cloudflare could cache the JSON version for an HTML page or vice-versa.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,
https://github.com/simonw/datasette/pull/1626#issuecomment-1028420821,https://api.github.com/repos/simonw/datasette/issues/1626,1028420821,IC_kwDOBm6k_c49THTV,9599,2022-02-02T22:32:26Z,2022-02-02T22:33:31Z,OWNER,"That broke on a macOS test: https://github.com/simonw/datasette/runs/5044036993?check_suite_focus=true
I'm going to remove macOS and Ubuntu and just try Windows purely to see what happens there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096,
https://github.com/simonw/datasette/pull/1616#issuecomment-1028414871,https://api.github.com/repos/simonw/datasette/issues/1616,1028414871,IC_kwDOBm6k_c49TF2X,9599,2022-02-02T22:23:45Z,2022-02-02T22:23:45Z,OWNER,First stable Black release!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1119413338,
https://github.com/simonw/datasette/issues/1623#issuecomment-1028397935,https://api.github.com/repos/simonw/datasette/issues/1623,1028397935,IC_kwDOBm6k_c49TBtv,9599,2022-02-02T21:59:43Z,2022-02-02T21:59:43Z,OWNER,Here's the new test: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/tests/test_html.py#L927-L936,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122416919,
https://github.com/simonw/datasette/issues/1624#issuecomment-1028396866,https://api.github.com/repos/simonw/datasette/issues/1624,1028396866,IC_kwDOBm6k_c49TBdC,9599,2022-02-02T21:58:06Z,2022-02-02T21:58:06Z,OWNER,"It looks like this is because `IndexView` extends `BaseView` rather than extending `DataView` which is where all that CORS stuff happens:
https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/datasette/views/index.py#L18-L21
Another thing I should address with the refactor project in:
- #878 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122427321,
https://github.com/simonw/datasette/issues/1620#issuecomment-1028393259,https://api.github.com/repos/simonw/datasette/issues/1620,1028393259,IC_kwDOBm6k_c49TAkr,9599,2022-02-02T21:53:02Z,2022-02-02T21:53:02Z,OWNER,"I ran the following on https://www.google.com/ in the console to demonstrate that these work as intended:
```javascript
[
""https://latest.datasette.io/fixtures"",
""https://latest.datasette.io/fixtures?sql=select+1"",
""https://latest.datasette.io/fixtures/facetable""
].forEach(async (url) => {
response = await fetch(url, {method: ""HEAD""});
console.log(response.headers.get(""Link""));
});
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,
https://github.com/simonw/datasette/issues/1623#issuecomment-1028389953,https://api.github.com/repos/simonw/datasette/issues/1623,1028389953,IC_kwDOBm6k_c49S_xB,9599,2022-02-02T21:48:34Z,2022-02-02T21:48:34Z,OWNER,"A few other pages do that too, including:
- https://latest.datasette.io/-/messages
- https://latest.datasette.io/-/allow-debug","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122416919,
https://github.com/simonw/datasette/issues/1620#issuecomment-1028385067,https://api.github.com/repos/simonw/datasette/issues/1620,1028385067,IC_kwDOBm6k_c49S-kr,9599,2022-02-02T21:42:23Z,2022-02-02T21:42:23Z,OWNER,"```
% curl -s -I 'https://latest.datasette.io/' | grep link
link: https://latest.datasette.io/.json; rel=""alternate""; type=""application/json+datasette""
% curl -s -I 'https://latest.datasette.io/fixtures' | grep link
link: https://latest.datasette.io/fixtures.json; rel=""alternate""; type=""application/json+datasette""
% curl -s -I 'https://latest.datasette.io/fixtures?sql=select+1' | grep link
link: https://latest.datasette.io/fixtures.json?sql=select+1; rel=""alternate""; type=""application/json+datasette""
% curl -s -I 'https://latest.datasette.io/-/plugins' | grep link
link: https://latest.datasette.io/-/plugins.json; rel=""alternate""; type=""application/json+datasette""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,
https://github.com/simonw/datasette/issues/1620#issuecomment-1028374330,https://api.github.com/repos/simonw/datasette/issues/1620,1028374330,IC_kwDOBm6k_c49S786,9599,2022-02-02T21:28:16Z,2022-02-02T21:28:16Z,OWNER,I just realized I can refactor this to make it much simpler.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,
https://github.com/simonw/datasette/issues/1533#issuecomment-1027672617,https://api.github.com/repos/simonw/datasette/issues/1533,1027672617,IC_kwDOBm6k_c49QQop,9599,2022-02-02T07:56:51Z,2022-02-02T07:56:51Z,OWNER,"Demos - these pages both have ` Table-valued functions exist only for PRAGMAs that return results and that have no side-effects.
So it's possible I'm being overly paranoid here after all: what I want to block here is people running things like `PRAGMA case_sensitive_like = 1` which could affect the global state for that connection and cause unexpected behaviour later on.
So maybe I should allow all pragma functions. I previously allowed an allow-list of them in:
- #761","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,
https://github.com/simonw/datasette/issues/1618#issuecomment-1027653005,https://api.github.com/repos/simonw/datasette/issues/1618,1027653005,IC_kwDOBm6k_c49QL2N,9599,2022-02-02T07:22:13Z,2022-02-02T07:22:13Z,OWNER,"There's a workaround for this at the moment, which is to use parameterized SQL queries. For example, this:
https://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+%3D+%3Atitle&title=The+Pragmatic+Programmer
So the SQL query is `select * from books where title = :title` and then `&title=...` is added to the URL.
The reason behind the quite aggressive pragma filtering is that SQLite allows you to execute pragmas using function calls, like this one:
```sql
SELECT * FROM pragma_index_info('idx52');
```
These can be nested arbitrarily deeply in sub-queries, so it's difficult to write a regular expression that will definitely catch them.
I'm open to relaxing the regex a bit, but I need to be very confident that it's safe to do so.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,
https://github.com/simonw/datasette/issues/1586#issuecomment-1027648180,https://api.github.com/repos/simonw/datasette/issues/1586,1027648180,IC_kwDOBm6k_c49QKq0,9599,2022-02-02T07:13:31Z,2022-02-02T07:13:31Z,OWNER,"Running it as part of `datasette publish` is a smart idea - I'm slightly nervous about modifying the database file that has been published though, since part of the undocumented contract right now is that the bytes served are the exact same bytes as the ones you ran the publish against.
But there's no reason for that expectation to exist, and I doubt anyone is relying on that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096536240,
https://github.com/simonw/datasette/issues/1619#issuecomment-1027647257,https://api.github.com/repos/simonw/datasette/issues/1619,1027647257,IC_kwDOBm6k_c49QKcZ,9599,2022-02-02T07:11:43Z,2022-02-02T07:11:43Z,OWNER,Weirdly the bug does NOT exhibit itself on this demo: https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1 - which correctly links to https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1.json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414,
https://github.com/simonw/datasette/issues/1619#issuecomment-1027646659,https://api.github.com/repos/simonw/datasette/issues/1619,1027646659,IC_kwDOBm6k_c49QKTD,9599,2022-02-02T07:10:37Z,2022-02-02T07:10:37Z,OWNER,It's not just the table with slashes in the name. Same thing on http://127.0.0.1:3344/foo/bar/fixtures/attraction_characteristic/1 - the `json` link goes to a JSON-rendered 404 on http://127.0.0.1:3344/foo/bar/foo/bar/fixtures/attraction_characteristic/1.json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414,
https://github.com/simonw/datasette/issues/1576#issuecomment-1027635925,https://api.github.com/repos/simonw/datasette/issues/1576,1027635925,IC_kwDOBm6k_c49QHrV,9599,2022-02-02T06:47:20Z,2022-02-02T06:47:20Z,OWNER,"Here's what I was hacking around with when I uncovered this problem:
```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 77fb285..8c57d08 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -1,3 +1,4 @@
+import asyncio
import urllib
import itertools
import json
@@ -615,44 +616,37 @@ class TableView(RowTableShared):
if request.args.get(""_timelimit""):
extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit""))
- # Execute the main query!
- results = await db.execute(sql, params, truncate=True, **extra_args)
-
- # Calculate the total count for this query
- filtered_table_rows_count = None
- if (
- not db.is_mutable
- and self.ds.inspect_data
- and count_sql == f""select count(*) from {table} ""
- ):
- # We can use a previously cached table row count
- try:
- filtered_table_rows_count = self.ds.inspect_data[database][""tables""][
- table
- ][""count""]
- except KeyError:
- pass
-
- # Otherwise run a select count(*) ...
- if count_sql and filtered_table_rows_count is None and not nocount:
- try:
- count_rows = list(await db.execute(count_sql, from_sql_params))
- filtered_table_rows_count = count_rows[0][0]
- except QueryInterrupted:
- pass
-
- # Faceting
- if not self.ds.setting(""allow_facet"") and any(
- arg.startswith(""_facet"") for arg in request.args
- ):
- raise BadRequest(""_facet= is not allowed"")
+ async def execute_count():
+ # Calculate the total count for this query
+ filtered_table_rows_count = None
+ if (
+ not db.is_mutable
+ and self.ds.inspect_data
+ and count_sql == f""select count(*) from {table} ""
+ ):
+ # We can use a previously cached table row count
+ try:
+ filtered_table_rows_count = self.ds.inspect_data[database][
+ ""tables""
+ ][table][""count""]
+ except KeyError:
+ pass
+
+ if count_sql and filtered_table_rows_count is None and not nocount:
+ try:
+ count_rows = list(await db.execute(count_sql, from_sql_params))
+ filtered_table_rows_count = count_rows[0][0]
+ except QueryInterrupted:
+ pass
+
+ return filtered_table_rows_count
+
+ filtered_table_rows_count = await execute_count()
# pylint: disable=no-member
facet_classes = list(
itertools.chain.from_iterable(pm.hook.register_facet_classes())
)
- facet_results = {}
- facets_timed_out = []
facet_instances = []
for klass in facet_classes:
facet_instances.append(
@@ -668,33 +662,58 @@ class TableView(RowTableShared):
)
)
- if not nofacet:
- for facet in facet_instances:
- (
- instance_facet_results,
- instance_facets_timed_out,
- ) = await facet.facet_results()
- for facet_info in instance_facet_results:
- base_key = facet_info[""name""]
- key = base_key
- i = 1
- while key in facet_results:
- i += 1
- key = f""{base_key}_{i}""
- facet_results[key] = facet_info
- facets_timed_out.extend(instance_facets_timed_out)
-
- # Calculate suggested facets
- suggested_facets = []
- if (
- self.ds.setting(""suggest_facets"")
- and self.ds.setting(""allow_facet"")
- and not _next
- and not nofacet
- and not nosuggest
- ):
- for facet in facet_instances:
- suggested_facets.extend(await facet.suggest())
+ async def execute_suggested_facets():
+ # Calculate suggested facets
+ suggested_facets = []
+ if (
+ self.ds.setting(""suggest_facets"")
+ and self.ds.setting(""allow_facet"")
+ and not _next
+ and not nofacet
+ and not nosuggest
+ ):
+ for facet in facet_instances:
+ suggested_facets.extend(await facet.suggest())
+ return suggested_facets
+
+ async def execute_facets():
+ facet_results = {}
+ facets_timed_out = []
+ if not self.ds.setting(""allow_facet"") and any(
+ arg.startswith(""_facet"") for arg in request.args
+ ):
+ raise BadRequest(""_facet= is not allowed"")
+
+ if not nofacet:
+ for facet in facet_instances:
+ (
+ instance_facet_results,
+ instance_facets_timed_out,
+ ) = await facet.facet_results()
+ for facet_info in instance_facet_results:
+ base_key = facet_info[""name""]
+ key = base_key
+ i = 1
+ while key in facet_results:
+ i += 1
+ key = f""{base_key}_{i}""
+ facet_results[key] = facet_info
+ facets_timed_out.extend(instance_facets_timed_out)
+
+ return facet_results, facets_timed_out
+
+ # Execute the main query, facets and facet suggestions in parallel:
+ (
+ results,
+ suggested_facets,
+ (facet_results, facets_timed_out),
+ ) = await asyncio.gather(
+ db.execute(sql, params, truncate=True, **extra_args),
+ execute_suggested_facets(),
+ execute_facets(),
+ )
+
+ results = await db.execute(sql, params, truncate=True, **extra_args)
# Figure out columns and rows for the query
columns = [r[0] for r in results.description]
```
It's a hacky attempt at running some of the table page queries in parallel to see what happens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,
https://github.com/simonw/datasette/issues/1611#issuecomment-1027635175,https://api.github.com/repos/simonw/datasette/issues/1611,1027635175,IC_kwDOBm6k_c49QHfn,9599,2022-02-02T06:45:47Z,2022-02-02T06:45:47Z,OWNER,"Prototype, not sure that this actually works yet:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 6ce8721..0c4aec7 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -256,18 +256,26 @@ class Database:
# Try to get counts for each table, $limit timeout for each count
counts = {}
for table in await self.table_names():
- try:
- table_count = (
- await self.execute(
- f""select count(*) from [{table}]"",
- custom_time_limit=limit,
- )
- ).rows[0][0]
- counts[table] = table_count
- # In some cases I saw ""SQL Logic Error"" here in addition to
- # QueryInterrupted - so we catch that too:
- except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError):
- counts[table] = None
+ print(table.lower())
+ if table.lower() == ""knn"":
+ counts[table] = 0
+ else:
+ try:
+ table_count = (
+ await self.execute(
+ f""select count(*) from [{table}]"",
+ custom_time_limit=limit,
+ )
+ ).rows[0][0]
+ counts[table] = table_count
+ # In some cases I saw ""SQL Logic Error"" here in addition to
+ # QueryInterrupted - so we catch that too:
+ except (
+ QueryInterrupted,
+ sqlite3.OperationalError,
+ sqlite3.DatabaseError,
+ ):
+ counts[table] = None
if not self.is_mutable:
self._cached_table_counts = counts
return counts
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1113384383,
https://github.com/simonw/datasette/issues/1607#issuecomment-1027634490,https://api.github.com/repos/simonw/datasette/issues/1607,1027634490,IC_kwDOBm6k_c49QHU6,9599,2022-02-02T06:44:30Z,2022-02-02T06:44:30Z,OWNER,"Prototype:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 09d7d03..e2a5aea 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -724,6 +724,47 @@ class Datasette:
sqlite_extensions[extension] = None
except Exception:
pass
+ # More details on SpatiaLite
+ if ""spatialite"" in sqlite_extensions:
+ spatialite_details = {}
+ fns = (
+ ""spatialite_version"",
+ ""spatialite_target_cpu"",
+ ""rcheck_strict_sql_quoting"",
+ ""freexl_version"",
+ ""proj_version"",
+ ""geos_version"",
+ ""rttopo_version"",
+ ""libxml2_version"",
+ ""HasIconv"",
+ ""HasMathSQL"",
+ ""HasGeoCallbacks"",
+ ""HasProj"",
+ ""HasProj6"",
+ ""HasGeos"",
+ ""HasGeosAdvanced"",
+ ""HasGeosTrunk"",
+ ""HasGeosReentrant"",
+ ""HasGeosOnlyReentrant"",
+ ""HasMiniZip"",
+ ""HasRtTopo"",
+ ""HasLibXML2"",
+ ""HasEpsg"",
+ ""HasFreeXL"",
+ ""HasGeoPackage"",
+ ""HasGCP"",
+ ""HasTopology"",
+ ""HasKNN"",
+ ""HasRouting"",
+ )
+ for fn in fns:
+ try:
+ result = conn.execute(""select {}()"".format(fn))
+ spatialite_details[fn] = result.fetchone()[0]
+ except Exception:
+ pass
+ sqlite_extensions[""spatialite""] = spatialite_details
+
# Figure out supported FTS versions
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109783030,
https://github.com/simonw/datasette/issues/1533#issuecomment-1027633686,https://api.github.com/repos/simonw/datasette/issues/1533,1027633686,IC_kwDOBm6k_c49QHIW,9599,2022-02-02T06:42:53Z,2022-02-02T06:42:53Z,OWNER,"I'm going to apply the hack, then fix it again in:
- #1518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,