issue_comments

Breaking this down into smaller steps:

• [x] Get ?_next= working
• [x] Implement extensions - so .json is needed again for the JSON version, and anything without an extension is passed through a new code path for HTML
• [ ] That HTML view should only access JSON data, which can be seen by using .context - this will require a lot of updates to templates (it may be necessary to still provide access to some helper functions though). This will form the basis of the ambition to fully document the template context.
• [ ] Get a bunch of the existing table HTML and JSON tests to pass
• [ ] Use those tests to refactor the nasty _next code, see https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469
• [ ] Figure out how the register_output_renderer(datasette) plugin hook should work

This feels quite nice:

I want to package together all of the extras that are needed for the HTML format. A few options for doing that:

• Introduce ?_extra=_html where the leading underscore indicates that this is a "bundle" of extras, then define a bundle that's everything needed for the HTML renderer
• Have some other mechanism whereby different renderers can request a bundle of extras.

I'm leaning towards the first option. I'll try that and see what it looks like.

To get this unblocked, I'm going to allow myself to pass non-JSON-serializable objects to the HTML template version of things. If I can get that working (and get the existing tests to pass) I can consider a later change that makes those JSON serializable - or admit that it's OK for the templates to have non-JSON data passed to them and figure out how best to document those variables independently from the JSON documentation.

The reason for that Row thing is that it allows custom templates that do things like this:

https://docs.datasette.io/en/stable/changelog.html#easier-custom-templates-for-table-rows

html+jinja {% for row in display_rows %} <div> <h2>{{ row["title"] }}</h2> <p>{{ row["description"] }}<lp> <p>Category: {{ row.display("category_id") }}</p> </div> {% endfor %} Is that a good design? the .display() thing feels weird - I wonder if anyone has ever actually used that.

It's documented here: https://docs.datasette.io/en/0.64.2/custom_templates.html#custom-templates

If you want to output the rendered HTML version of a column, including any links to foreign keys, you can use {{ row.display("column_name") }}.

I can't see any examples of anyone using it in this code search: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+row.display

It is however useful to have some kind of abstraction layer here that insulates the SQLite Row object, since having an extra layer will help if Datasette ever grows support for alternative database backends such as DuckDB or PostgreSQL.

Aside idea: it might be interesting if there were "lazy" template variables available in the context: things that are not actually executed unless a template author requests them.

Imagine if metadata was a lazy template reference, such that custom templates that don't display any metadata don't trigger it to be resolved (which might involve additional database queries some day).

Figuring out what to do with display_columns_and_rows() is hard. That returns rows as this special kind of object, which is designed to be accessed from the HTML templates:

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L45-L71

For the HTML version, I need to decide where all of the stuff that happens in async def extra_template() is going to live.

I think it's another one of those extra functions, triggered for ?_extra=context.

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L813-L912

The ease with which I added that ?_extra=query feature in https://github.com/simonw/datasette/pull/1999/commits/96e94f9b7b2db53865e61390bcce6761727f26d8 made me feel really confident that this architecture is going in the right direction.

```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 8d3bb2c930..3e1db9c85f 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1913,6 +1913,13 @@ async def extra_request(): "args": request.args._data, }

• async def extra_query():
• "Details of the underlying SQL query"
• return {
• "sql": sql,
• "params": params,
• } + async def extra_extras(): "Available ?_extra= blocks" return { @@ -1938,6 +1945,7 @@ async def extra_extras(): extra_primary_keys, extra_debug, extra_request,
• extra_query, extra_extras, ) ```

Just noticed that _json=colname is not working, and that's because it's handled by the renderer here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/renderer.py#L29-L40

But that's not currently being called by my new code.

I'm going to hold off on that refactor until later, when I have tests to show me if the refactor works or not.

So maybe I can refactor it to look a bit more like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1602-L1604

One thing that's useful here is that is_view is handled early, like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L466-L472

So if I omit the is_view bit from the extracted function I can simplify more.

I'd really like to extract this ugly logic out into a helper function:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1617-L1684

I copied it in and asked ChatGPT to "List all of the variable that are either modified or assigned to by the above code":

Modified variables:

offset
where_clauses
params
order_by

Assigned variables:

sort_value
next_by_pk_clauses
order_by (when sort or sort_desc is not None)

Then I asked which variables were used as inputs, and argued with it a bit about whether it should be counting functions. Eventually got to this:

My apologies for the oversight. Here are the variables needed as input by the above code, excluding any functions:

_next
is_view
sort
sort_desc
use_rowid
pks
order_by

Note that use_rowid is actually defined earlier in terms of two of those other variables: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1540

I'm trying to get http://127.0.0.1:8001/fixtures/compound_three_primary_keys?_next=a,d,v to return the correct results.