issue_comments

As noted in https://github.com/simonw/datasette/issues/621#issuecomment-552253208 a common pattern in this method is blocks of code that append new items to the where_clauses, params and extra_human_descriptions arrays. This is a useful refactoring opportunity.

Code that fits this pattern:

• The code that builds based on the filters: where_clauses, params = filters.build_where_clauses(table) and human_description_en = filters.human_description_en(extra=extra_human_descriptions)
• Code that handles ?_where=: where_clauses.extend(request.args["_where"]) - though note that this also appends to a extra_wheres_for_ui array which nothing else uses
• The _through= code, see #621 for details
• The code that deals with ?_search= FTS

The keyset pagination code modifies where_clauses and params too, but I don't think it's quite going to work with the same abstraction that would cover the above examples.

[UPDATE December 2021 - this comment became the basis for a new filters_from_request plugin hook, see also #473]

This work is now happening in #1518

I made some comments on an annotated version of this method a few days ago: https://gist.github.com/simonw/66aca058195d77bae5f614ef73352eb5

That table_filter() plugin hook should probably be renamed though, since it could now apply to the regular custom SQL view as well as the table view.

This is what the code does:

• Check if the table name is actually a canned query - if so execute that
• Check for other parameters that need to be redirected - e.g. ?_filter_column=x or _sort_by_desc=y
• Turn the table + querystring parameters (?foo=bar, ?bar__contains=baz, ?_where=, _sort= etc) into a SQL query
• Figure out pagination (apply limit, and handle keyset pagination)
• Calculate suggested facets against the non-paginated version of the query
• Calculate facet counts against the non-paginated version of the query
• Execute the query
• Expand any foreign key references from the results

I experimented with a table_filter() plugin hook a while ago which looks very much like the abstraction I'm now talking about here: https://github.com/simonw/datasette/commit/5116c4ec8aed5091e1f75415424b80f613518dc6 - more details here: https://simonwillison.net/2018/Aug/6/russian-facebook-ads/#Weird_implementation_details_106

The function that builds the query could go in a new datasette.utils.sql module.

I can design it to only take simple arguments (the table name, list of columns, list of primary keys and a list of key/value tuples from the query string). This will make it really easy to test (and means it won't need to be async since it won't have to use queries to retrieve those values).

A clean starting point would be to refactor out the "take a table and a bunch of querystring parameters and turn them into a SQL query" portion.

Added bonus to this: I've long wanted to be able to apply the various trimmings of the TableView (like faceting and foreign key expansion) to other arbitrary SQL queries. Splitting out the code that builds the SELECT query will go a long way to making that a reality.

Here's the current monstrosity: https://github.com/simonw/datasette/blob/f9c146b893856a48afa810ebcce1714f30d0d3a2/datasette/views/table.py#L210-L780