issue_comments

This filter design can only influence the where component of the SQL clause - it's not able to modify the SELECT columns or adjust the ORDER BY or OFFSET LIMIT parts. I think that's OK.

Got a TIL out of this: https://til.simonwillison.net/pluggy/multiple-hooks-same-file

I revisited this idea in #1518 and came up with a slightly different name and design for the hook:

python @hookspec def filters_from_request(request, database, table, datasette): """ Return FilterArguments( where_clauses=[str, str, str], params={}, human_descriptions=[str, str, str], extra_context={} ) based on the request"""

I managed to extract both _search= and _where= out using a prototype of that hook. I wonder if it could extract the complex code for ?_next too?

Ran into a problem prototyping that hook up for handling ?_where= - that feature also adds a little bit of extra template context in order to show the interface for removing wheres - the extra_wheres_for_ui variable: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L457-L463

Maybe change to this?

python class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] extra_context: Dict[str, Any] That might be necessary for _search too.

The hook could return a named tuple like this one: ```python from typing import NamedTuple, List, Optional, Union, Dict

class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] ```

I think that plugin hook would get given the request object (and datasette and the name of the database and table) and returns a list of SQL fragments, a dictionary of lookup arguments and a list of human-description fragments - or an awaitable.

filters_from_request(request, database, table, datasette) perhaps? (Similar in name to actor_from_request).

python @hookspec def filters_from_request(request, database, table, datasette): """Return (where_clauses, params_dict, human_descriptions) based on the request"""

Turns out that's pretty much exactly what I implemented in 5116c4ec8aed5091e1f75415424b80f613518dc6 for #473:

```python

@hookspec def table_filter(): "Custom filtering of the current table based on the request" python TableFilter = namedtuple("TableFilter", ( "human_description_extras", "where_clauses", "params") ) python # filter_arguments plugin hook support for awaitable_fn in pm.hook.table_filter(): extras = await awaitable_fn( view=self, name=name, table=table, request=request ) human_description_extras.extend(extras.human_description_extras) where_clauses.extend(extras.where_clauses) params.update(extras.params) ```

This goes beyond the transform() method - the curious methods that create new SQL tables could benefit from the ability to add CHECK constraints too.

I haven't used these myself, do you have any CREATE TABLE examples that use them that you can share?

Wow, that was fast, thank you so much @simonw !

I'm also not convinced that this configuration syntax is right. It's a bit weird having a "facets" list that can either by column-name-strings or {"type-of-facet": "column-name"} objects. Maybe there's a better design for this?

I agree that it's not ideal, my initial naive approach was to detect if it's an array, like what is done here:

https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/facets.py#L312-L313

But it requires an extra query to determine the type, which is a bit problematic, especially for big tables I guess.

Taking a look at #510, I wonder if a facet_delimiter should be defined for that kind of columns (that would help our team not to have an intermediary conversion step from foo|bar to ["foo","bar"] for instance).

To be consistent with the --extract-column parameter, maybe an explicit casting/delimiter would be useful: --set-column 'Foo:Array:|'.

Throwing a lot of ideas without knowing the big picture… but sometimes newcomers have superpowers :).

Is there an opportunity to refactor things using a new plugin hook here? Maybe the register_filters hook from #473, where the hook becomes responsible for building where clauses (and human descriptions of them) based on the incoming query string.

That version dealt with Filter classes, but those might be a bit too low-level for this.

?_spatial_within=GEOJSON was an interesting idea attached to that issue.

I added a ton of comments to the data() method which really helps get a better feel for how this all works: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L322

A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters.

The human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says:

1 row where search matches "dog" and text2 = "sara weasel" sorted by pk

(Also used in the page <title>).

The code actually gathers three things:

• Fragments of the where clause, for example "text2" = :p0
• Parameters, e.g. {"p0": "sara weasel"}
• Human description components, e.g. text2 = "sara weasel"

Some operations such as ?_where= don't currently provide an extra human description component.

_where= also doesn't populate a parameter, but maybe it could? Would be neat if in the future ?_where=foo+=+:bar worked and added a bar input field to the screen, as seen with custom queries.

Should facets really not be displayed on pages past page one (where ?_next= is set)? That made sense to me at the time, but I'm now having second thoughts about it.

I guess it's a useful performance tweak for when crawlers keep hitting the ?_next= link.

Actually it looks like facets DO display on subsequent pages, e.g. on https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 - but facet suggestions do not, thanks to this code: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/views/table.py#L777-L785

Related: Following the fix for #625 I noticed that facets_timed_out gives you just the column name, but doesn't let you know which particular type of facet (date or array for example) suffered the timeout: https://github.com/simonw/datasette/blob/0d4145d0f4d8b2a7edc1ba4aac1be56cd536a10a/datasette/facets.py#L269-L270

Thanks!

And here's the new JSON: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1

{ "database": "fixtures", "table": "facetable", "is_view": false, "human_description_en": "", ... "facet_results": { "created": { "name": "created", "type": "column", ... }, "tags": { "name": "tags", "type": "column", ... }, "created_2": { "name": "created", "type": "date", ... }, "tags_2": { "name": "tags", "type": "array", ... } } }

Demo of the fix: https://latest.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags#facet-tags

I updated the example code in the facet plugin hook documentation: https://github.com/simonw/datasette/blob/95d0dd7a1cf6be6b7da41e1404184217eb93f64a/docs/plugin_hooks.rst#register_facet_classes

Since no-one is using that plugin hook I'm going to alter its contract slightly. I'll still keep the existing JSON format working though (until 1.0), since it's much more likely that people are using that JSON somewhere.

As of today, 16 December 2021, I'm still not seeing any evidence that anyone is using this hook (yet) according to GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette

Good news - GitHub's new code search doesn't show ANYONE using that plugin hook - not surprising since it has that documentation warning plus it's just not a very clearly usable hook: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette

This bad design is even covered in the plugin hooks documentation: https://docs.datasette.io/en/0.59.4/plugin_hooks.html#register-facet-classes

It does at least have the following warning:

Warning

The design of this plugin hook is unstable and may change. See issue 830.

Here's where facet_results is built up: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L752-L758

So the decision to key things based on column name is actually embedded deep in the existing facet classes here:

https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L224-L226

https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L395-L397

https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L510-L512

The flaw in the current design is illustrated by this example:

"facet_results": { "tags": { "name": "tags", "type": "array", "results": [...], "hideable": false, "toggle_url": "/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1", "truncated": false }, "created": { "name": "created", "type": "date", "results": [...] This was the cause of the bug in #625 - the each of those objects is keyed by the name of the column, which left no room for faceting the same column once by date and once by column value.

Decision: as an initial fix I'm going to de-duplicate those keys by using tags__array etc - with a _2 on the end if that key is already used.

I'll open a separate issue to redesign this better for Datasette 1.0.

Really facet_results here should be an array of objects, not an object that maps poorly designed string keys to those objects.

Datasette's own HTML rendering code doesn't actually use the keys in facet_results - it instead loops through sorted_facet_results which is defined like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L937-L941

And used like this:

https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/templates/table.html#L154-L156

The big question here is do I break any existing clients of the "facet_results" JSON API?

It's still pre-1.0 so I could break them, but I've also built my own code against this in the past so it's likely other people have too.

If I don't break them, I will instead need to come up with a naming convention for those keys - something like "tags__array" for example. As well as a way to ensure that a column called tags__array doesn't end up conflicting with the tags__array key!

Demo: compare https://latest.datasette.io/fixtures/facetable?_facet=_city_id&_nosuggest=1 to https://latest.datasette.io/fixtures/facetable?_facet=_city_id

Documentation: bottom of https://docs.datasette.io/en/latest/json_api.html#special-table-arguments

Demo: https://latest.datasette.io/fixtures/facetable?_facet=planet_int&_facet=_city_id&_facet=created#facet-created

This is a really interesting idea - kind of similar to how many APIs include custom HTTP headers informing of rate-limits.

I think the problem here may be in the design of the JSON returned by facets. It looks like this: "facet_results": { "tags": { "name": "tags", "type": "array", "results": [...], "hideable": false, "toggle_url": "/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1", "truncated": false }, "created": { "name": "created", "type": "date", "results": [...] The problem then is that the tags key is over-ridden by the second facet with a different type against the same column name!

https://latest-with-plugins.datasette.io/fixtures/facetable?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags confirms that the SQL queries for those facets are being executed - but the final JSON doesn't show them on https://latest-with-plugins.datasette.io/fixtures/facetable.json?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags

They're not available in the template context either: https://latest-with-plugins.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet_array=tags&_facet=tags&_context=1

Implementing #1552 has made a fix for this bug even more important.

Updated documentation: https://github.com/simonw/datasette/blob/20a2ed6bec367d2f6759be4a879364a72780b59d/docs/facets.rst#facets-in-metadatajson

... actually no, I WILL document this, because not documenting this is what got us to this point in the first place!

I'm going to ship your fix now, but I'm not going to add this to the documentation yet because I hope to improve the design prior to Datasette 1.0.

I'm also not convinced that this configuration syntax is right. It's a bit weird having a "facets" list that can either by column-name-strings or {"type-of-facet": "column-name"} objects. Maybe there's a better design for this?

Part of the problem here is that facets were designed to accept optional extra configuration - partly to support m2m facets in #495 - but I haven't actually shipped any facets that use that ability.

Facet by delimiter would be a good one to exercise that ability: - #510

I tried that fix you suggested and now this metadata.json does the right thing: json { "databases": { "fixtures": { "tables": { "facetable": { "facets": [ { "array": "tags" } ] } } } } } It does further highlight the bug in #625 though - since then if you try to add ?_facet=tags to facet by tags treating them NOT as an array your request to do so is ignored.

I think you're right! I had completely forgotten that piece of code.

This just turned into a bug fix and a documentation update. Thanks for the research!