issue_comments

Relevant code: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/init.py#L919-L940

In that case maybe there are three new arguments: path, full_path and url.

I'll also add request.full_path for consistency with these: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/asgi.py#L77-L90

I think I'll call this full_path for consistency with Django.

Django calls this HttpRequest.get_full_path() - for the path plus the querystring.

This parameter will return the URL path, with querystring arguments, to the HTML version of the page - e.g. /github/issue_comments or /github/issue_comments?_sort_desc=created_at

Open questions:

• What should it be called? path could be misleading since it also includes the querystring.
• Should I provide a url or full_url version which includes https://blah.com/...?

In building https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on I discovered the following patterns for importing data into both Pandas and Observable/d3: python import pandas df = pandas.read_json( "https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array" ) And: javascript d3 = require("d3@5") rows = d3.json( "https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array" ) Once again I find myself torn on the best possible default. A list of JSON objects is instantly compatible with both pandas.read_json() and d3.json() - but it leaves nowhere to put the extra information like pagination and suchlike!

Even given this I still think the correct default is an object with "rows", "total" and "next_url" keys. I should commit to that and implement it - this thought exercise has been running for far too long.

Issue fixed - https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on displays the correct schemes now.

I can't think of a reason anyone on Cloud Run would ever NOT want the force_https_urls option, but just in case I've made it so if you pass --extra-options --setting force_https_urls off to publish cloudrun your setting will be respected.

https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/publish/cloudrun.py#L105-L110

Deploying that change now to test it.

Easiest fix would be for publish cloudrun to set force_https_urls:

datasette publish now used to do this: https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/publish/now.py#L59-L63

An optional path argument to https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette which shows the path WITHOUT the .Notebook extension would be useful here.

https://latest-with-plugins.datasette.io/-/asgi-scope

{'asgi': {'spec_version': '2.1', 'version': '3.0'}, 'client': ('169.254.8.129', 54971), 'headers': [(b'host', b'latest-with-plugins.datasette.io'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko' b'/20100101 Firefox/84.0'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'webp,*/*;q=0.8'), (b'accept-language', b'en-US,en;q=0.5'), (b'dnt', b'1'), (b'cookie', b'_ga_LL6M7BK6D4=GS1.1.1609886546.49.1.1609886923.0; _ga=GA1.1' b'.894633707.1607575712'), (b'upgrade-insecure-requests', b'1'), (b'x-client-data', b'CgSL6ZsV'), (b'x-cloud-trace-context', b'e776af843c657d2a3da28a73b726e6fe/14187666787557102189;o=1'), (b'x-forwarded-for', b'148.64.98.14'), (b'x-forwarded-proto', b'https'), (b'forwarded', b'for="148.64.98.14";proto=https'), (b'accept-encoding', b'gzip, deflate, br'), (b'content-length', b'0')], 'http_version': '1.1', 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', 'root_path': '', 'scheme': 'http', 'server': ('169.254.8.130', 8080), 'type': 'http'} Note the 'scheme': 'http' but also the (b'x-forwarded-proto', b'https').

I used from datasette.utils import path_with_format in https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/init.py just now.

Moving this to the Datasette repo.

https://latest-with-plugins.datasette.io/fixtures/sortable.json has the bug too - the next_url is http:// when it should be https://.

https://latest-with-plugins.datasette.io/-/settings says "force_https_urls": false

Weird... https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/app.py#L609-L613

python def absolute_url(self, request, path): url = urllib.parse.urljoin(request.url, path) if url.startswith("http://") and self.setting("force_https_urls"): url = "https://" + url[len("http://") :] return url That looks like it should work. Needs more digging.

https://github.com/simonw/datasette-export-notebook/blob/aec398eab4f34791d240d7bc47b6eec575b357be/datasette_export_notebook/init.py#L18-L23

Maybe this is a bug in datasette.absolute_url? Perhaps it doesn't take the scheme into account.

With this structure it will become possible to stream non-newline-delimited JSON array-of-objects too - the stream_rows() method could output [ first, then each row followed by a comma, then ] after the very last row.

Idea: instead of returning a dictionary, register_output_renderer could return an object. The object could have the following properties:

• .extension - the extension to use
• .can_render(...) - says if it can render this
• .can_stream(...) - says if streaming is supported
• async .stream_rows(rows_iterator, send) - method that loops through all rows and uses send to send them to the response in the correct format

I can then deprecate the existing dict return type for 1.0.

Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:

pandas.read_json("https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on", lines=True)