{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392343839", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392343839, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjM0MzgzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T16:10:09Z", "updated_at": "2018-06-04T17:38:04Z", "author_association": "OWNER", "body": "The more efficient way of doing this kind of count would be to provide a mechanism which can also add extra fragments to a `GROUP BY` clause used for the `SELECT`.\r\n\r\nOr... how about a mechanism similar to Django's `prefetch_related` which lets you define extra queries that will be called with a list of primary keys (or values from other columns) and used to populate a new column? A little unconventional but could be extremely useful and efficient.\r\n\r\nRelated to that: since the per-query overhead in SQLite is tiny, could even define an extra query to be run once-per-row before returning results.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392350980", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392350980, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjM1MDk4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T17:56:30Z", "updated_at": "2018-05-27T17:56:50Z", "author_association": "OWNER", "body": "Should `?_raw=1` also turn off foreign key expansions? No, we will eventually provide a separate mechanism for that (or leave it to nerds who care to figure out using JSON or CSV export).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392350568", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392350568, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjM1MDU2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T17:48:45Z", "updated_at": "2018-05-27T17:54:41Z", "author_association": "OWNER", "body": "If any `?_column=` parameters are provided the metadata version is completely ignored.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392350495", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392350495, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjM1MDQ5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T17:47:31Z", "updated_at": "2018-05-27T17:47:31Z", "author_association": "OWNER", "body": "Querystring design:\r\n\r\n* `?_column=a&_column=b` - equivalent of `\"columns\": [\"a\", \"b\"]` in `metadata.json`\r\n* `?_select_nameupper=upper(name)` - equivalent of `\"column_selects\": {\"nameupper\": \"upper(name)\"}`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392345062", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392345062, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjM0NTA2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T16:26:53Z", "updated_at": "2018-05-27T16:26:53Z", "author_association": "OWNER", "body": "There needs to be a way to turn this off and return to Datasette default bahviour. Maybe a `?_raw=1` querystring parameter for the table view.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392343690", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392343690, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjM0MzY5MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T16:08:25Z", "updated_at": "2018-05-27T16:08:40Z", "author_association": "OWNER", "body": "Turns out it's actually possible to pull data from other tables using the mechanism in the prototype:\r\n\r\n```\r\n{\r\n \"databases\": {\r\n \"wtr\": {\r\n \"tables\": {\r\n \"license\": {\r\n \"column_selects\": {\r\n \"count\": \"(select count(*) from license_frequency where license_frequency.license = license.id)\"\r\n }\r\n }\r\n }\r\n }\r\n }\r\n}\r\n```\r\nPerformance using this technique is pretty terrible though:\r\n\r\n\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392342947", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392342947, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjM0Mjk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T16:01:43Z", "updated_at": "2018-05-27T16:01:43Z", "author_association": "OWNER", "body": "I'd still like to be able to over-ride this using querystring arguments.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392342269", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392342269, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjM0MjI2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T15:55:40Z", "updated_at": "2018-05-27T16:01:26Z", "author_association": "OWNER", "body": "Here's the metadata I tried against that first working prototype:\r\n\r\n```\r\n{\r\n \"databases\": {\r\n \"timezones\": {\r\n \"tables\": {\r\n \"timezones\": {\r\n \"columns\": [\"PK_UID\"],\r\n \"column_selects\": {\r\n \"upper_tzid\": \"upper(tzid)\",\r\n \"Geometry\": \"AsGeoJSON(Geometry)\"\r\n }\r\n }\r\n }\r\n },\r\n \"wtr\": {\r\n \"tables\": {\r\n \"license_frequency\": {\r\n \"columns\": [\"id\", \"license\", \"tx_rx\", \"frequency\"],\r\n \"column_selects\": {\r\n \"latitude\": \"Y(Geometry)\",\r\n \"longitude\": \"X(Geometry)\"\r\n }\r\n }\r\n }\r\n }\r\n }\r\n}\r\n```\r\n\r\nRun using this:\r\n\r\n datasette timezones.db wtr.db \\\r\n --reload --debug --load-extension=/usr/local/lib/mod_spatialite.dylib \\\r\n -m column-metadata.json --config sql_time_limit_ms:10000\r\n\r\nUsefully, the `--reload` flag detects changes to the `metadata.json` file as well as Datasette's own Python code.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392338130", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392338130, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjMzODEzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T15:09:18Z", "updated_at": "2018-05-27T15:09:28Z", "author_association": "OWNER", "body": "Here's my first sketch at a metadata format for this:\r\n\r\n* `columns`: optional list of columns to include - if missing, shows all\r\n* `column_selects`: dictionary mapping column names to alternative select clauses\r\n\r\n`column_selects` can also invent new keys and use them to create derived columns. These new keys will be selected at the end of the list of columns UNLESS they are mentioned in `columns`, in which case that sequence will define the order.\r\n\r\nCan you facet by things that are customized using `column_selects`? Yes, and let's try running suggested facets against those columns as well.\r\n```\r\n{\r\n \"databases\": {\r\n \"databasename\": {\r\n \"tables\": {\r\n \"tablename\": {\r\n \"columns\": [\r\n \"id\", \"name\", \"size\"\r\n ],\r\n \"column_selects\": {\r\n \"name\": \"upper(name)\",\r\n \"geo_json\": \"AsGeoJSON(Geometry)\"\r\n }\r\n \"row_columns\": [...]\r\n \"row_column_selects\": {...}\r\n }\r\n```\r\nThe `row_columns` and `row_column_selects` properties work the same as the `column*` ones, except they are applied on the row page instead.\r\n\r\nIf omitted, the `column*` ones will be used on the row page as well.\r\n\r\nIf you want the row page to switch back to Datasette's default behaviour you can set `\"row_columns\": [], \"row_column_selects\": {}`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392316701", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392316701, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjMxNjcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T09:08:49Z", "updated_at": "2018-05-27T09:08:49Z", "author_association": "OWNER", "body": "I could certainly see people wanting different custom column selects for the row page compared to the table page.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/292#issuecomment-392316673", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/292", "id": 392316673, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjMxNjY3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T09:08:06Z", "updated_at": "2018-05-27T09:08:06Z", "author_association": "OWNER", "body": "Open question: how should this affect the row page? Just because columns were hidden on the table page doesn't necessarily mean they should be hidden on the row page as well. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326800219, "label": "Mechanism for customizing the SQL used to select specific columns in the table view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/276#issuecomment-392316306", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/276", "id": 392316306, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjMxNjMwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T09:00:46Z", "updated_at": "2018-05-27T09:00:46Z", "author_association": "OWNER", "body": "Relevant to this ticket: I've been playing with a plugin that automatically renders any GeoJSON cells as leaflet maps: https://github.com/simonw/datasette-leaflet-geojson", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 324835838, "label": "Handle spatialite geometry columns better"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/276#issuecomment-392316250", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/276", "id": 392316250, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjMxNjI1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T08:59:46Z", "updated_at": "2018-05-27T08:59:46Z", "author_association": "OWNER", "body": "It looks like we can use the `geometry_columns` table to introspect which columns are SpatiaLite geometries. It includes a `geometry_type` integer which is documented here: https://www.gaia-gis.it/fossil/libspatialite/wiki?name=switching-to-4.0", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 324835838, "label": "Handle spatialite geometry columns better"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/231#issuecomment-392305776", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/231", "id": 392305776, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjMwNTc3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T05:10:46Z", "updated_at": "2018-05-27T05:10:46Z", "author_association": "OWNER", "body": "These plugin config options should be exposed to JavaScript as `datasette.config.plugins`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 316323336, "label": "metadata.json support for plugin configuration options"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/291#issuecomment-392302456", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/291", "id": 392302456, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjMwMjQ1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T03:19:24Z", "updated_at": "2018-05-27T03:19:24Z", "author_association": "OWNER", "body": "The big gap in this solution is conflicting versions: I don't yet have a story for what happens if two plugins attempt to load different versions of Leaflet. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326783670, "label": "Avoid plugins accidentally loading dependencies twice"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/291#issuecomment-392302416", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/291", "id": 392302416, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjMwMjQxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T03:18:16Z", "updated_at": "2018-05-27T03:18:16Z", "author_association": "OWNER", "body": "For the moment then I'm going with a really simple solution: when iterating through `extra_css_urls` and `extra_js_urls` de-dupe by URL and avoid outputting the same link twice.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326783670, "label": "Avoid plugins accidentally loading dependencies twice"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/291#issuecomment-392302406", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/291", "id": 392302406, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjMwMjQwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T03:18:06Z", "updated_at": "2018-05-27T03:18:06Z", "author_association": "OWNER", "body": "My first attempt at this was to have plugins depend on each other - so there would be a `datasette-leaflet` plugin which adds Leaflet to the page, and the `datasette-cluster-map` and `datasette-leaflet-geojson` plugins would depend on that plugin.\r\n\r\nI tried this and it didn't work, because it turns out the order in which plugins are loaded isn't predictable. `datasette-cluster-map` ended up adding it's script link before Leaflet had been loaded by `datasette-leaflet`, resulting in JavaScript errors.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326783670, "label": "Avoid plugins accidentally loading dependencies twice"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/285#issuecomment-392297508", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/285", "id": 392297508, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjI5NzUwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T00:53:35Z", "updated_at": "2018-05-27T00:53:35Z", "author_association": "OWNER", "body": "Documentation: http://datasette.readthedocs.io/en/latest/config.html#num-sql-threads", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326189744, "label": "num_threads and cache_max_age should be --config options"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/285#issuecomment-392297392", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/285", "id": 392297392, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjI5NzM5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T00:50:27Z", "updated_at": "2018-05-27T00:50:27Z", "author_association": "OWNER", "body": "I ran a very rough micro-benchmark on the new `num_sql_threads` config option.\r\n\r\n datasette --config num_sql_threads:1 fivethirtyeight.db\r\n\r\nThen\r\n\r\n ab -n 100 -c 10 'http://127.0.0.1:8011/fivethirtyeight-2628db9/twitter-ratio%2Fsenators'\r\n\r\n| Number of threads | Requests/second |\r\n|---|---|\r\n| 1 | 4.57 |\r\n| 3 | 9.77 |\r\n| 10 | 13.53 |\r\n| 20 | 15.24 \r\n| 50 | 8.21 | \r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326189744, "label": "num_threads and cache_max_age should be --config options"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/287#issuecomment-392296758", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/287", "id": 392296758, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MjI5Njc1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-27T00:32:53Z", "updated_at": "2018-05-27T00:32:53Z", "author_association": "OWNER", "body": "Docs: https://datasette.readthedocs.io/en/latest/json_api.html#different-shapes", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 326617744, "label": "?_shape=arrayfirst"}, "performed_via_github_app": null}