issue_comments

https://github.com/simonw/datasette/compare/facet-refactor-2 is almost ready to merge now. The remaining things to do are listed as TODOs there:

• [x] Ensure facet is not suggested if it is already active
• [x] Don't allow facets to be hidden if they were configured in metadata.json

It would be neat to ship at least one additional face with this work - probably either ArrayFacet or DateFacet. I think ArrayFacet because it demonstrates the only-if-json1-enabled functionality.

I think there's a Facet base class.

class ColumnFacet(Facet): is the default behaviour we have today

class ArrayFacet(Facet): facet by JSON array

class ManyToManyFacet(Facet): facet by M2M table

class DateFacet(Facet): facet by date

class DateTimeFacet(Facet): facet by datetime

class EmojiFacet(Facet): super-fun demo plugin I have planned

Could even have a facet against a numerical column which loads the entire set of column values into numpy or pandas and calculates complex statistics facets in memory .

There’s actually a lot of potential for Datasette plugins that load several MBs of data and analyze using other Python libraries.

A facet needs to: - given a sql query and a list of configs, return a list of buckets - Know how to generate URLs for selecting and deselecting a filter (along with underlying filter application sql logic) - Tell if a specific filter is currently selected or not - Set a time limit and report if it times out - Generate human readable labels - In some cases: expand foreign keys - which means they need access to foreign key information - just the name of the table and the name of the column is enough to call expand_foreign_keys() (I moved that to the Datasette class to make it easier to access) - Make suggestions for facets. Let's give it access to the whole table here so it could either run against each column in return and rely with a list of suggestions or it could spot eg a latitude and a longitude column

TableView.data is currently the longest, hairiest method in the codebase. It's 775 - 177 = 598 lines of code! Extracting faceting logic should help reduce that quite a bit.

https://github.com/simonw/datasette/blob/274ef43bb7b129ddc2e68805b4f4ff3776fb9503/datasette/views/table.py#L177-L775

Bonus idea: since we are having a Facet abstraction we should allow additional facet type apps to be registered using a plugin.

Fun idea for a (very inefficient) demo plugin: facet-by-emoji! Would work by counting all emoji in text fields using a horrible slow full-scan regular expression, then would apply selected emoji facets using a LIKE query.

I just realized: a key part of faceting is being able to correctly apply the facet (and know that it has been applied).

Existing facets are exact match only, so they can be applied and detected with ?foo=bar

More advanced facets like _facet_array and _facet_m2m will need different ways of applying themselves. This needs to be bundled up in the new Facet abstraction somehow.

This means the metadata.json format can look like this: { "databases": { "sf-trees": { "tables": { "Street_Tree_List": { "facets": ["qLegalStatus", {"array": "tags"}, {"percentile": {"blah": "options"}}] } } } } }

So any advanced facets are represented here as a dictionary with a single key - the type - that maps to the options.

OK, I have a plan:

?_facet=foo ?_facet_facettype=options

Options here can be one of the following:

• A single value which is the name of a table
• A comma separated list of options
• A JSON object starting with { or [

If the table name itself contains a ,, { or ] then you have to escape it by putting it in a JSON object, ?_facet_percentile={"column":"{this_is,a_weird[column_name"} for example.

I quite like the Solr idea. It could look like this for Datasette:

?_facet=name - default behaviour, same as today. But that's actually an alias for ?_facet.name=name - which defines a name for the facet.

?_facet.tags.array=tags - would define a facet called tags that uses an array facet against the tags column.

I don't like the need to say tags twice in that though.