issue_comments

The more efficient way of doing this kind of count would be to provide a mechanism which can also add extra fragments to a GROUP BY clause used for the SELECT.

Or... how about a mechanism similar to Django's prefetch_related which lets you define extra queries that will be called with a list of primary keys (or values from other columns) and used to populate a new column? A little unconventional but could be extremely useful and efficient.

Related to that: since the per-query overhead in SQLite is tiny, could even define an extra query to be run once-per-row before returning results.

Should ?_raw=1 also turn off foreign key expansions? No, we will eventually provide a separate mechanism for that (or leave it to nerds who care to figure out using JSON or CSV export).

If any ?_column= parameters are provided the metadata version is completely ignored.

Querystring design:

• ?_column=a&_column=b - equivalent of "columns": ["a", "b"] in metadata.json
• ?_select_nameupper=upper(name) - equivalent of "column_selects": {"nameupper": "upper(name)"}

There needs to be a way to turn this off and return to Datasette default bahviour. Maybe a ?_raw=1 querystring parameter for the table view.

Turns out it's actually possible to pull data from other tables using the mechanism in the prototype:

{ "databases": { "wtr": { "tables": { "license": { "column_selects": { "count": "(select count(*) from license_frequency where license_frequency.license = license.id)" } } } } } } Performance using this technique is pretty terrible though:

I'd still like to be able to over-ride this using querystring arguments.

Here's the metadata I tried against that first working prototype:

{ "databases": { "timezones": { "tables": { "timezones": { "columns": ["PK_UID"], "column_selects": { "upper_tzid": "upper(tzid)", "Geometry": "AsGeoJSON(Geometry)" } } } }, "wtr": { "tables": { "license_frequency": { "columns": ["id", "license", "tx_rx", "frequency"], "column_selects": { "latitude": "Y(Geometry)", "longitude": "X(Geometry)" } } } } } }

Run using this:

datasette timezones.db wtr.db \
    --reload --debug --load-extension=/usr/local/lib/mod_spatialite.dylib \
    -m column-metadata.json --config sql_time_limit_ms:10000

Usefully, the --reload flag detects changes to the metadata.json file as well as Datasette's own Python code.

Here's my first sketch at a metadata format for this:

• columns: optional list of columns to include - if missing, shows all
• column_selects: dictionary mapping column names to alternative select clauses

column_selects can also invent new keys and use them to create derived columns. These new keys will be selected at the end of the list of columns UNLESS they are mentioned in columns, in which case that sequence will define the order.

Can you facet by things that are customized using column_selects? Yes, and let's try running suggested facets against those columns as well. { "databases": { "databasename": { "tables": { "tablename": { "columns": [ "id", "name", "size" ], "column_selects": { "name": "upper(name)", "geo_json": "AsGeoJSON(Geometry)" } "row_columns": [...] "row_column_selects": {...} } The row_columns and row_column_selects properties work the same as the column* ones, except they are applied on the row page instead.

If omitted, the column* ones will be used on the row page as well.

If you want the row page to switch back to Datasette's default behaviour you can set "row_columns": [], "row_column_selects": {}.

I could certainly see people wanting different custom column selects for the row page compared to the table page.

Open question: how should this affect the row page? Just because columns were hidden on the table page doesn't necessarily mean they should be hidden on the row page as well.