issue_comments

For the overall shape of the rows:

?_shape=lists (default), ?_shape=objects, ?_shape=object (primary key as object keys)

For getting back extra keys:

?_extras=schema,query,timing

For expanding columns:

?_expand_all=1

Or ?_expand=qSpecies&_expand=qCaretaker

The template view will only be allowed to work with data it can request using extra options.

That leaves one sighted nasty edge-case: the default view will expand all columns, but the .json view of it won't?

I think that's OK. The default view won't include the extras used by the template to render the page either.

I need a nicer abstraction around the concept of filters. It needs to be able to:

• convert querystring parameters into filters
• convert filters into a querystring
• iterate through currently applied filters
• convert selected filters into a human description (e.g. for a title)
• expand filters that involve a foreign key
• add filters
• remove filters
• define different types of filters

It should replace my current build_where_clauses implementation, in particular this bit:

https://github.com/simonw/datasette/blob/a5881e105a02830d26f07e98177248d5910893da/datasette/utils.py#L38-L56

To start with, I could just ditch the .jsono in favour of the new _shape argument.

This calls for refactoring the code so the table view, the row view and the custom SQL view share as much logic as possible.

I implemented a basic version of this in f59c840e7db8870afcdeba7a53bdea07bb674334 for custom SQL.

This would enable faceted search - moving it to the search milestone.

Dupe of #127

Demo: https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List?_search=grove+st

Since SQLite supports column specifications in the MATCH body itself, there's no need to provide a separate mechanism for specifying columns in the query string: https://sqlite.org/fts5.html#fts5_column_filters

Relevant SQLite docs:

• https://sqlite.org/fts5.html
• https://www.sqlite.org/fts3.html

Specifically docs should make it clearer this file exists

https://parlgov.datasettes.com/.json

And from that you can build https://parlgov.datasettes.com/parlgov-25f9855.json

Then https://parlgov.datasettes.com/parlgov-25f9855/cabinet.json

Thanks, I wrote this very simple reader because the default approach as described on the Datahub pages seemed to complicated. I had metadata from the datapackage.json attached to the returned DataFrames but removed this due to some attribute handling change in the latest Pandas version.

This could also be useful for getting from Data Package to SQL db: https://github.com/frictionlessdata/tableschema-sql-py

I maintain a few climate science related dataset at https://github.com/openclimatedata/

The Data Retriever (mainly ecological data) by @ethanwhite et al. is also using the Data Package format for metadata and has some tooling for different dbs:

https://frictionlessdata.io/articles/the-data-retriever/ https://github.com/weecology/retriever

The Open Power System Data project also has a couple of datasets that show nicely how CSV is great for assembling and then already make SQLite files available. It's one of the first data sets I tried with Datasette, perfect for the use case of getting an API for putting power stations on a map ...

https://data.open-power-system-data.org/

I'll have to refactor the foreign key annotating code to be usable in other contexts - at the moment it only works for annotating displays of rows, but I need to use it to resolve selected filters as well.

"Tablename: 3,567 rows where status = 3 (published) and n > 55"

I'm going to be a bit classier about this and auto generate a title for the page that describes the currently applied filters.

OK,I've figured out how to do an initial version of this without JavaScript. I'll provide three form fields labell d "add filter":

• a select box of all of the columns
• a select box of the available operations
• a value box

Submit those and the site will redirect you to a correctly populated querystring for that filter.

If you have filters applied, those will display as prepopulated form field triples.

For foreign key reference filters, I will display the resolved value next to the text box containing the numeric ID. In the future this can get a select2 style treatment.

Maybe I should support &_count=1 to handle this - that would be easy to Ajax-in in conjenction with the other filters.

It would be great if this could support foreign key references and automatically resolve and hyperlink them if they are detected.

If the selected relationship is a foreign key reference, we should resolve that foreign key and display it on the page.

Now that we have foreign key support (#85) this is even more important, since foreign key support actively encourages linking to filtered table views.

This is working really nicely now:

https://github.com/rgieseke/pandas-datapackage-reader

Looks like there are a ton of interesting datasets packaged in this way at http://datahub.io/docs/core-data - see also https://github.com/datasets