{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-398098582", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 398098582, "node_id": "MDEyOklzc3VlQ29tbWVudDM5ODA5ODU4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-18T15:40:32Z", "updated_at": "2018-06-18T15:40:32Z", "author_association": "OWNER", "body": "This is now released in Datasette 0.23! http://datasette.readthedocs.io/en/latest/changelog.html#v0-23", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397952129", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397952129, "node_id": "MDEyOklzc3VlQ29tbWVudDM5Nzk1MjEyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-18T06:15:36Z", "updated_at": "2018-06-18T06:15:51Z", "author_association": "OWNER", "body": "Advanced export pane demo: https://latest.datasette.io/fixtures-35b6eb6/facetable?_size=4", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397842667", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397842667, "node_id": "MDEyOklzc3VlQ29tbWVudDM5Nzg0MjY2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-16T22:38:15Z", "updated_at": "2018-06-18T05:55:11Z", "author_association": "OWNER", "body": "Still todo:\r\n\r\n- [x] Streaming version\r\n- [ ] Tidy up the \"This data as ...\" UI\r\n- [x] Default .csv (and .json) links to use `?_labels=on` (only if at least one foreign key detected)\r\n\r\n\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397949002", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397949002, "node_id": "MDEyOklzc3VlQ29tbWVudDM5Nzk0OTAwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-18T05:53:17Z", "updated_at": "2018-06-18T05:53:17Z", "author_association": "OWNER", "body": "Advanced export pane:\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397923253", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397923253, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzkyMzI1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-18T01:49:52Z", "updated_at": "2018-06-18T03:02:28Z", "author_association": "OWNER", "body": "Ideally the downloadable filenames of exported CSVs would differ across different querystring parameters. Maybe S`treet_Trees-56cbd54.csv` where `56cbd54` is a hash of the querystring?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397918264", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397918264, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzkxODI2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-18T00:49:35Z", "updated_at": "2018-06-18T00:49:35Z", "author_association": "OWNER", "body": "Simpler design: the top of the page will link to basic .json and .csv and \"advanced\" - which will fragment link to an advanced export format the bottom of the page.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397916321", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397916321, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzkxNjMyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-18T00:17:44Z", "updated_at": "2018-06-18T00:18:05Z", "author_association": "OWNER", "body": "The export UI could be a GET form controlling various parameters. This would discourage crawlers from hitting the export links and would also allow us to express the full range of export options.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397916091", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397916091, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzkxNjA5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-18T00:13:43Z", "updated_at": "2018-06-18T00:15:50Z", "author_association": "OWNER", "body": "I was also worried about the performance of pagination over custom `_sort` orders or views which use offset pagination - but Datasette's SQL time limits should prevent those from getting out of hand. This does mean that a streaming CSV file may be truncated with an error - if this happens we should ensure the error is written out as the last line of the CSV so anyone who tried to import it gets a relevant error message informing them that the export did not complete.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397915403", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397915403, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzkxNTQwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-18T00:03:17Z", "updated_at": "2018-06-18T00:14:37Z", "author_association": "OWNER", "body": "Since CSV streaming export doesn't work for custom SQL queries (since they don't support `_next=` pagination) there's no need to provide a option that disables streams just for custom SQL.\r\n\r\nRelated: the UI should not show the option to download everything on custom SQL pages.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397915258", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397915258, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzkxNTI1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-18T00:01:05Z", "updated_at": "2018-06-18T00:01:05Z", "author_association": "OWNER", "body": "Someone malicious could use a UNION to generate an unpleasantly large CSV response. I'll add another config setting which limits the response size to 100MB but can be turned off by setting it to 0.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397912840", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397912840, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzkxMjg0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-17T23:13:35Z", "updated_at": "2018-06-17T23:16:42Z", "author_association": "OWNER", "body": "This worked! https://github.com/simonw/datasette/commit/5a0a82faf9cf9dd109d76181ed00eea19472087c - it spat out a 76MB CSV when I ran it against the sf-trees demo database.\r\n\r\nIt was just a quick hack though - it currently ignores `_labels=` and `_dl=` which need to be supported.\r\n\r\nI'm going to add a config option for turning full CSV export off just in case any Datasette users are uncomfortable with URLs that churn out that much data in one go.\r\n\r\n```\r\n ConfigOption(\"allow_csv_stream\", True, \"\"\"\r\n Allow .csv?_stream=1 to download all rows (ignoring max_returned_rows)\r\n \"\"\".strip()),\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397842246", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397842246, "node_id": "MDEyOklzc3VlQ29tbWVudDM5Nzg0MjI0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-16T22:27:59Z", "updated_at": "2018-06-16T22:27:59Z", "author_association": "OWNER", "body": "Two demos of the new functionality in #233 as it applies to CSV:\r\n\r\n* https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List.csv?_labels=on - CSV with all foreign key columns expanded\r\n* https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List.csv?_label=qSpecies - CSV with specific columns expanded", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397729945", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397729945, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzcyOTk0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-15T20:13:05Z", "updated_at": "2018-06-15T20:13:05Z", "author_association": "OWNER", "body": "The \"This data as ...\" area of the page is getting a bit untidy, especially if I'm going to add other download options in the future. I think I'll move the HTML to the page footer (less concerns about taking up lots of space there) and then have a bit of JavaScript that turns it into a show/hide menu of some sort in its current location.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-389570841", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 389570841, "node_id": "MDEyOklzc3VlQ29tbWVudDM4OTU3MDg0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-16T15:54:49Z", "updated_at": "2018-06-15T07:41:09Z", "author_association": "OWNER", "body": "At the most basic level, this will work based on an extension. Most places you currently put a `.json` extension should also allow a `.csv` extension.\r\n\r\nBy default this will return the exact results you see on the current page (default max will remain 1000).\r\n\r\n## Streaming all records\r\n\r\nWhere things get interested is *streaming mode*. This will be an option which returns ALL matching records as a streaming CSV file, even if that ends up being millions of records.\r\n\r\nI think the best way to build this will be on top of the existing mechanism used to efficiently implement keyset pagination via `_next=` tokens.\r\n\r\n## Expanding foreign keys\r\n\r\nFor tables with foreign key references it would be useful if the CSV format could expand those references to include the labels from `label_column` - maybe via an additional `?_expand=1` option.\r\n\r\nWhen expanding each foreign key column will be shown twice:\r\n\r\n rowid,city_id,city_id_label,state", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397534498", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397534498, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzUzNDQ5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-15T07:13:52Z", "updated_at": "2018-06-15T07:13:52Z", "author_association": "OWNER", "body": "Also needs documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397534404", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397534404, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzUzNDQwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-15T07:13:20Z", "updated_at": "2018-06-15T07:13:20Z", "author_association": "OWNER", "body": "Still to add: the streaming version that iterates through all of the pages, as seen in experimental commit https://github.com/simonw/datasette/commit/ced379ea325787b8c3bf0a614daba1fa4856a3bd", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-397534196", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 397534196, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NzUzNDE5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-15T07:12:16Z", "updated_at": "2018-06-15T07:12:16Z", "author_association": "OWNER", "body": "The first version of this is now shipped to master. I ended up rewriting most of the experimental branch to deal with the nasty issue described in #303\r\n\r\nDemo is available on https://fivethirtyeight.datasettes.com/fivethirtyeight-ab24e01/most-common-name%2Fsurnames\r\n\r\n\r\n\r\nHere's the CSV version of that page: https://fivethirtyeight.datasettes.com/fivethirtyeight-ab24e01/most-common-name%2Fsurnames.csv", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-394417567", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 394417567, "node_id": "MDEyOklzc3VlQ29tbWVudDM5NDQxNzU2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-06-04T16:30:48Z", "updated_at": "2018-06-04T16:32:55Z", "author_association": "OWNER", "body": "When serving streaming responses, I need to check that a large CSV file doesn't completely max out the CPU in a way that is harmful to the rest of the instance.\r\n\r\nIf it does, one option may be to insert an async sleep call in between each chunk that is streamed back. This could be controlled by a `csv_pause_ms` config setting, defaulting to maybe 5 but can be disabled entirely by setting to 0.\r\n\r\nThat's only if testing proves that this is a necessary mechanism.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-393020749", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 393020749, "node_id": "MDEyOklzc3VlQ29tbWVudDM5MzAyMDc0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-30T03:42:54Z", "updated_at": "2018-05-30T03:42:54Z", "author_association": "OWNER", "body": "Challenge: how to deal with tables where the name ends in `.csv`? I actually have one of these in the test suite at the moment:\r\n\r\nhttps://github.com/simonw/datasette/blob/d69ebce53385b7c6fafb85fdab3b136dbf3f332c/tests/fixtures.py#L234-L237", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-389894382", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 389894382, "node_id": "MDEyOklzc3VlQ29tbWVudDM4OTg5NDM4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-17T14:51:13Z", "updated_at": "2018-05-17T14:53:23Z", "author_association": "OWNER", "body": "I should definitely sanity check if the `_next=` route really is the most efficient way to build this. It may turn out that iterating over a SQLite cursor with a million rows in it is super-efficient and would provide much more reliable performance (plus solve the problem for retrieving full custom SQL queries where we can't do keyset pagination).\r\n\r\nProblem here is that we run SQL queries in a thread pool. A query that returns millions of rows would presumably tie up a SQL thread until it has finished, which could block the server. This may be a reason to stick with `_next=` keyset pagination - since it ensures each SQL thread yields back again after each 1,000 rows.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-389893810", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 389893810, "node_id": "MDEyOklzc3VlQ29tbWVudDM4OTg5MzgxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-17T14:49:35Z", "updated_at": "2018-05-17T14:49:35Z", "author_association": "OWNER", "body": "Idea: add a `supports_csv = False` property to `BaseView` and over-ride it to `True` just on the view classes that should support CSV (Table and Row). Slight subtlety: the `DatabaseView` class only supports CSV in the `custom_sql()` path. Maybe that needs to be refactored a bit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-389626715", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 389626715, "node_id": "MDEyOklzc3VlQ29tbWVudDM4OTYyNjcxNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-16T18:50:46Z", "updated_at": "2018-05-16T18:50:46Z", "author_association": "OWNER", "body": "> I\u2019d recommend using the Windows-1252 encoding for maximum compatibility, unless you have any characters not in that set, in which case use UTF8 with a byte order mark. Bit of a pain, but some progams (eg various versions of Excel) don\u2019t read UTF8.\r\n**frankieroberto** https://twitter.com/frankieroberto/status/996823071947460616\r\n\r\n> There is software that consumes CSV and doesn't speak UTF8!? Huh. Well I can't just use Windows-1252 because I need to support the full UTF8 range of potential data - maybe I should support an optional ?_encoding=windows-1252 argument\r\n**simonw** https://twitter.com/simonw/status/996824677245857793", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-389608473", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 389608473, "node_id": "MDEyOklzc3VlQ29tbWVudDM4OTYwODQ3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-16T17:52:35Z", "updated_at": "2018-05-16T17:54:11Z", "author_association": "OWNER", "body": "There are some code examples in this issue which should help with the streaming part: https://github.com/channelcat/sanic/issues/1067\r\n\r\nAlso https://github.com/channelcat/sanic/blob/master/docs/sanic/streaming.md#response-streaming", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-389592566", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 389592566, "node_id": "MDEyOklzc3VlQ29tbWVudDM4OTU5MjU2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-16T17:01:29Z", "updated_at": "2018-05-16T17:02:21Z", "author_association": "OWNER", "body": "Let's provide a CSV Dialect definition too: https://frictionlessdata.io/specs/csv-dialect/ - via https://twitter.com/drewdaraabrams/status/996794915680997382", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-389579762", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 389579762, "node_id": "MDEyOklzc3VlQ29tbWVudDM4OTU3OTc2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-16T16:21:12Z", "updated_at": "2018-05-16T16:21:12Z", "author_association": "OWNER", "body": "> I basically want someone to tell me which arguments I can pass to Python's csv.writer() function that will result in the least complaints from people who try to parse the results :)\r\nhttps://twitter.com/simonw/status/996786815938977792", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-389579363", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 389579363, "node_id": "MDEyOklzc3VlQ29tbWVudDM4OTU3OTM2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-16T16:20:06Z", "updated_at": "2018-05-16T16:20:06Z", "author_association": "OWNER", "body": "I started a thread on Twitter discussing various CSV output dialects: https://twitter.com/simonw/status/996783395504979968 - I want to pick defaults which will work as well as possible for whatever tools people might be using to consume the data.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/266#issuecomment-389572201", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/266", "id": 389572201, "node_id": "MDEyOklzc3VlQ29tbWVudDM4OTU3MjIwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-05-16T15:58:43Z", "updated_at": "2018-05-16T16:00:47Z", "author_association": "OWNER", "body": "This will likely be implemented in the `BaseView` class, which needs to know how to spot the `.csv` extension, call the underlying JSON generating function and then return the `columns` and `rows` as correctly formatted CSV.\r\n\r\nhttps://github.com/simonw/datasette/blob/9959a9e4deec8e3e178f919e8b494214d5faa7fd/datasette/views/base.py#L201-L207\r\n\r\nThis means it will take ALL arguments that are available to the `.json` view. It may ignore some (e.g. `_facet=` makes no sense since CSV tables don't have space to show the facet results).\r\n\r\nIn streaming mode, things will behave a little bit differently - in particular, if `_stream=1` then `_next=` will be forbidden.\r\n\r\nIt can't include a length header because we don't know how many bytes it will be\r\n\r\nCSV output will throw an error if the endpoint doesn't have rows and columns keys eg `/-/inspect.json`\r\n\r\nSo the implementation...\r\n\r\n- looks for the `.csv` extension\r\n- internally fetches the `.json` data instead\r\n- If no `_stream` it just transposes that JSON to CSV with the correct content type header\r\n- If `_stream=1` - checks for `_next=` and throws an error if it was provided\r\n- Otherwise... fetch first page and emit CSV header and first set of rows\r\n- Then start async looping, emitting more CSV rows and following the `_next=` internal reference until done\r\n\r\nI like that this takes advantage of efficient pagination. It may not work so well for views which use offset/limit though.\r\n\r\nIt won't work at all for custom SQL because custom SQL doesn't support _next= pagination. That's fine.\r\n\r\nFor views... easiest fix is to cut off after first X000 records. That seems OK. View JSON would need to include a property that the mechanism can identify.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323681589, "label": "Export to CSV"}, "performed_via_github_app": null}