issue_comments

27 rows where issue = 323681589 sorted by updated_at descending

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

user

issue

  • Export to CSV · 27

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
398098582 https://github.com/simonw/datasette/issues/266#issuecomment-398098582 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5ODA5ODU4Mg== simonw 9599 2018-06-18T15:40:32Z 2018-06-18T15:40:32Z OWNER

This is now released in Datasette 0.23! http://datasette.readthedocs.io/en/latest/changelog.html#v0-23

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397952129 https://github.com/simonw/datasette/issues/266#issuecomment-397952129 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5Nzk1MjEyOQ== simonw 9599 2018-06-18T06:15:36Z 2018-06-18T06:15:51Z OWNER

Advanced export pane demo: https://latest.datasette.io/fixtures-35b6eb6/facetable?_size=4

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397842667 https://github.com/simonw/datasette/issues/266#issuecomment-397842667 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5Nzg0MjY2Nw== simonw 9599 2018-06-16T22:38:15Z 2018-06-18T05:55:11Z OWNER

Still todo:

  • Streaming version
  • Tidy up the "This data as ..." UI
  • Default .csv (and .json) links to use ?_labels=on (only if at least one foreign key detected)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397949002 https://github.com/simonw/datasette/issues/266#issuecomment-397949002 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5Nzk0OTAwMg== simonw 9599 2018-06-18T05:53:17Z 2018-06-18T05:53:17Z OWNER

Advanced export pane:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397923253 https://github.com/simonw/datasette/issues/266#issuecomment-397923253 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzkyMzI1Mw== simonw 9599 2018-06-18T01:49:52Z 2018-06-18T03:02:28Z OWNER

Ideally the downloadable filenames of exported CSVs would differ across different querystring parameters. Maybe Street_Trees-56cbd54.csv where 56cbd54 is a hash of the querystring?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397918264 https://github.com/simonw/datasette/issues/266#issuecomment-397918264 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzkxODI2NA== simonw 9599 2018-06-18T00:49:35Z 2018-06-18T00:49:35Z OWNER

Simpler design: the top of the page will link to basic .json and .csv and "advanced" - which will fragment link to an advanced export format the bottom of the page.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397916321 https://github.com/simonw/datasette/issues/266#issuecomment-397916321 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzkxNjMyMQ== simonw 9599 2018-06-18T00:17:44Z 2018-06-18T00:18:05Z OWNER

The export UI could be a GET form controlling various parameters. This would discourage crawlers from hitting the export links and would also allow us to express the full range of export options.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397916091 https://github.com/simonw/datasette/issues/266#issuecomment-397916091 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzkxNjA5MQ== simonw 9599 2018-06-18T00:13:43Z 2018-06-18T00:15:50Z OWNER

I was also worried about the performance of pagination over custom _sort orders or views which use offset pagination - but Datasette's SQL time limits should prevent those from getting out of hand. This does mean that a streaming CSV file may be truncated with an error - if this happens we should ensure the error is written out as the last line of the CSV so anyone who tried to import it gets a relevant error message informing them that the export did not complete.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397915403 https://github.com/simonw/datasette/issues/266#issuecomment-397915403 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzkxNTQwMw== simonw 9599 2018-06-18T00:03:17Z 2018-06-18T00:14:37Z OWNER

Since CSV streaming export doesn't work for custom SQL queries (since they don't support _next= pagination) there's no need to provide a option that disables streams just for custom SQL.

Related: the UI should not show the option to download everything on custom SQL pages.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397915258 https://github.com/simonw/datasette/issues/266#issuecomment-397915258 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzkxNTI1OA== simonw 9599 2018-06-18T00:01:05Z 2018-06-18T00:01:05Z OWNER

Someone malicious could use a UNION to generate an unpleasantly large CSV response. I'll add another config setting which limits the response size to 100MB but can be turned off by setting it to 0.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397912840 https://github.com/simonw/datasette/issues/266#issuecomment-397912840 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzkxMjg0MA== simonw 9599 2018-06-17T23:13:35Z 2018-06-17T23:16:42Z OWNER

This worked! https://github.com/simonw/datasette/commit/5a0a82faf9cf9dd109d76181ed00eea19472087c - it spat out a 76MB CSV when I ran it against the sf-trees demo database.

It was just a quick hack though - it currently ignores _labels= and _dl= which need to be supported.

I'm going to add a config option for turning full CSV export off just in case any Datasette users are uncomfortable with URLs that churn out that much data in one go.

    ConfigOption("allow_csv_stream", True, """
        Allow .csv?_stream=1 to download all rows (ignoring max_returned_rows)
    """.strip()),
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397842246 https://github.com/simonw/datasette/issues/266#issuecomment-397842246 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5Nzg0MjI0Ng== simonw 9599 2018-06-16T22:27:59Z 2018-06-16T22:27:59Z OWNER

Two demos of the new functionality in #233 as it applies to CSV:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397729945 https://github.com/simonw/datasette/issues/266#issuecomment-397729945 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzcyOTk0NQ== simonw 9599 2018-06-15T20:13:05Z 2018-06-15T20:13:05Z OWNER

The "This data as ..." area of the page is getting a bit untidy, especially if I'm going to add other download options in the future. I think I'll move the HTML to the page footer (less concerns about taking up lots of space there) and then have a bit of JavaScript that turns it into a show/hide menu of some sort in its current location.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
389570841 https://github.com/simonw/datasette/issues/266#issuecomment-389570841 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM4OTU3MDg0MQ== simonw 9599 2018-05-16T15:54:49Z 2018-06-15T07:41:09Z OWNER

At the most basic level, this will work based on an extension. Most places you currently put a .json extension should also allow a .csv extension.

By default this will return the exact results you see on the current page (default max will remain 1000).

Streaming all records

Where things get interested is streaming mode. This will be an option which returns ALL matching records as a streaming CSV file, even if that ends up being millions of records.

I think the best way to build this will be on top of the existing mechanism used to efficiently implement keyset pagination via _next= tokens.

Expanding foreign keys

For tables with foreign key references it would be useful if the CSV format could expand those references to include the labels from label_column - maybe via an additional ?_expand=1 option.

When expanding each foreign key column will be shown twice:

rowid,city_id,city_id_label,state
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397534498 https://github.com/simonw/datasette/issues/266#issuecomment-397534498 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzUzNDQ5OA== simonw 9599 2018-06-15T07:13:52Z 2018-06-15T07:13:52Z OWNER

Also needs documentation.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397534404 https://github.com/simonw/datasette/issues/266#issuecomment-397534404 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzUzNDQwNA== simonw 9599 2018-06-15T07:13:20Z 2018-06-15T07:13:20Z OWNER

Still to add: the streaming version that iterates through all of the pages, as seen in experimental commit https://github.com/simonw/datasette/commit/ced379ea325787b8c3bf0a614daba1fa4856a3bd

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
397534196 https://github.com/simonw/datasette/issues/266#issuecomment-397534196 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NzUzNDE5Ng== simonw 9599 2018-06-15T07:12:16Z 2018-06-15T07:12:16Z OWNER

The first version of this is now shipped to master. I ended up rewriting most of the experimental branch to deal with the nasty issue described in #303

Demo is available on https://fivethirtyeight.datasettes.com/fivethirtyeight-ab24e01/most-common-name%2Fsurnames

Here's the CSV version of that page: https://fivethirtyeight.datasettes.com/fivethirtyeight-ab24e01/most-common-name%2Fsurnames.csv

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
394417567 https://github.com/simonw/datasette/issues/266#issuecomment-394417567 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5NDQxNzU2Nw== simonw 9599 2018-06-04T16:30:48Z 2018-06-04T16:32:55Z OWNER

When serving streaming responses, I need to check that a large CSV file doesn't completely max out the CPU in a way that is harmful to the rest of the instance.

If it does, one option may be to insert an async sleep call in between each chunk that is streamed back. This could be controlled by a csv_pause_ms config setting, defaulting to maybe 5 but can be disabled entirely by setting to 0.

That's only if testing proves that this is a necessary mechanism.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
393020749 https://github.com/simonw/datasette/issues/266#issuecomment-393020749 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM5MzAyMDc0OQ== simonw 9599 2018-05-30T03:42:54Z 2018-05-30T03:42:54Z OWNER

Challenge: how to deal with tables where the name ends in .csv? I actually have one of these in the test suite at the moment:

https://github.com/simonw/datasette/blob/d69ebce53385b7c6fafb85fdab3b136dbf3f332c/tests/fixtures.py#L234-L237

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
389894382 https://github.com/simonw/datasette/issues/266#issuecomment-389894382 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM4OTg5NDM4Mg== simonw 9599 2018-05-17T14:51:13Z 2018-05-17T14:53:23Z OWNER

I should definitely sanity check if the _next= route really is the most efficient way to build this. It may turn out that iterating over a SQLite cursor with a million rows in it is super-efficient and would provide much more reliable performance (plus solve the problem for retrieving full custom SQL queries where we can't do keyset pagination).

Problem here is that we run SQL queries in a thread pool. A query that returns millions of rows would presumably tie up a SQL thread until it has finished, which could block the server. This may be a reason to stick with _next= keyset pagination - since it ensures each SQL thread yields back again after each 1,000 rows.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
389893810 https://github.com/simonw/datasette/issues/266#issuecomment-389893810 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM4OTg5MzgxMA== simonw 9599 2018-05-17T14:49:35Z 2018-05-17T14:49:35Z OWNER

Idea: add a supports_csv = False property to BaseView and over-ride it to True just on the view classes that should support CSV (Table and Row). Slight subtlety: the DatabaseView class only supports CSV in the custom_sql() path. Maybe that needs to be refactored a bit.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
389626715 https://github.com/simonw/datasette/issues/266#issuecomment-389626715 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM4OTYyNjcxNQ== simonw 9599 2018-05-16T18:50:46Z 2018-05-16T18:50:46Z OWNER

I’d recommend using the Windows-1252 encoding for maximum compatibility, unless you have any characters not in that set, in which case use UTF8 with a byte order mark. Bit of a pain, but some progams (eg various versions of Excel) don’t read UTF8.
frankieroberto https://twitter.com/frankieroberto/status/996823071947460616

There is software that consumes CSV and doesn't speak UTF8!? Huh. Well I can't just use Windows-1252 because I need to support the full UTF8 range of potential data - maybe I should support an optional ?_encoding=windows-1252 argument
simonw https://twitter.com/simonw/status/996824677245857793

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
389608473 https://github.com/simonw/datasette/issues/266#issuecomment-389608473 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM4OTYwODQ3Mw== simonw 9599 2018-05-16T17:52:35Z 2018-05-16T17:54:11Z OWNER

There are some code examples in this issue which should help with the streaming part: https://github.com/channelcat/sanic/issues/1067

Also https://github.com/channelcat/sanic/blob/master/docs/sanic/streaming.md#response-streaming

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
389592566 https://github.com/simonw/datasette/issues/266#issuecomment-389592566 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM4OTU5MjU2Ng== simonw 9599 2018-05-16T17:01:29Z 2018-05-16T17:02:21Z OWNER

Let's provide a CSV Dialect definition too: https://frictionlessdata.io/specs/csv-dialect/ - via https://twitter.com/drewdaraabrams/status/996794915680997382

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
389579762 https://github.com/simonw/datasette/issues/266#issuecomment-389579762 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM4OTU3OTc2Mg== simonw 9599 2018-05-16T16:21:12Z 2018-05-16T16:21:12Z OWNER

I basically want someone to tell me which arguments I can pass to Python's csv.writer() function that will result in the least complaints from people who try to parse the results :)
https://twitter.com/simonw/status/996786815938977792

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
389579363 https://github.com/simonw/datasette/issues/266#issuecomment-389579363 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM4OTU3OTM2Mw== simonw 9599 2018-05-16T16:20:06Z 2018-05-16T16:20:06Z OWNER

I started a thread on Twitter discussing various CSV output dialects: https://twitter.com/simonw/status/996783395504979968 - I want to pick defaults which will work as well as possible for whatever tools people might be using to consume the data.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589
389572201 https://github.com/simonw/datasette/issues/266#issuecomment-389572201 https://api.github.com/repos/simonw/datasette/issues/266 MDEyOklzc3VlQ29tbWVudDM4OTU3MjIwMQ== simonw 9599 2018-05-16T15:58:43Z 2018-05-16T16:00:47Z OWNER

This will likely be implemented in the BaseView class, which needs to know how to spot the .csv extension, call the underlying JSON generating function and then return the columns and rows as correctly formatted CSV.

https://github.com/simonw/datasette/blob/9959a9e4deec8e3e178f919e8b494214d5faa7fd/datasette/views/base.py#L201-L207

This means it will take ALL arguments that are available to the .json view. It may ignore some (e.g. _facet= makes no sense since CSV tables don't have space to show the facet results).

In streaming mode, things will behave a little bit differently - in particular, if _stream=1 then _next= will be forbidden.

It can't include a length header because we don't know how many bytes it will be

CSV output will throw an error if the endpoint doesn't have rows and columns keys eg /-/inspect.json

So the implementation...

  • looks for the .csv extension
  • internally fetches the .json data instead
  • If no _stream it just transposes that JSON to CSV with the correct content type header
  • If _stream=1 - checks for _next= and throws an error if it was provided
  • Otherwise... fetch first page and emit CSV header and first set of rows
  • Then start async looping, emitting more CSV rows and following the _next= internal reference until done

I like that this takes advantage of efficient pagination. It may not work so well for views which use offset/limit though.

It won't work at all for custom SQL because custom SQL doesn't support _next= pagination. That's fine.

For views... easiest fix is to cut off after first X000 records. That seems OK. View JSON would need to include a property that the mechanism can identify.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Export to CSV 323681589

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 31.897ms · About: github-to-sqlite