issue_comments

Table actions

40 rows where "created_at" is on date 2020-10-20 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, reactions, created_at (date), updated_at (date)

user 3

author_association 2

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
712988146 https://github.com/simonw/datasette/issues/262#issuecomment-712988146 https://api.github.com/repos/simonw/datasette/issues/262 MDEyOklzc3VlQ29tbWVudDcxMjk4ODE0Ng== simonw 9599 2020-10-20T16:32:02Z 2023-01-17T01:54:13Z OWNER

Just realized I added an undocumented ?_extras= option to the row view years ago and forgot about it - it's not even documented. Added in a30c5b220c15360d575e94b0e67f3255e120b916 - https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables

That will need to be made consistent with the new mechanism. I think ?_extra=a&_extra=b is more consistent with other Datasette features (like ?_facet=col1&_facet=col2) but potentially quite verbose.

So I could support ?_extra=a,b,c as an alternative allowed syntax, or I could allow ?_extra=single and ?_extras=comma,separated.

I think I prefer allowing commas in ?_extra=.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add ?_extra= mechanism for requesting extra properties in JSON 323658641  
713200782 https://github.com/simonw/datasette/issues/262#issuecomment-713200782 https://api.github.com/repos/simonw/datasette/issues/262 MDEyOklzc3VlQ29tbWVudDcxMzIwMDc4Mg== simonw 9599 2020-10-20T23:41:30Z 2020-10-20T23:41:30Z OWNER

This is now blocking https://github.com/simonw/datasette-graphql/issues/61 because that issue needs a way to turn off suggested facets when retrieving the results of a table query.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add ?_extra= mechanism for requesting extra properties in JSON 323658641  
713191819 https://github.com/simonw/datasette/issues/1034#issuecomment-713191819 https://api.github.com/repos/simonw/datasette/issues/1034 MDEyOklzc3VlQ29tbWVudDcxMzE5MTgxOQ== simonw 9599 2020-10-20T23:12:58Z 2020-10-20T23:12:58Z OWNER

Enzo has a great solution here: https://twitter.com/enzo_mdd/status/1318685442976436226

Or maybe an option for a url. This keeps the CSV small but allows scripts to download binary data as needed.

In #1036 I'm planning on adding a way for users to access BLOB data. I can include that URL in the CSV output.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Better way of representing binary data in .csv output 725184645  
713186189 https://github.com/simonw/datasette/issues/1036#issuecomment-713186189 https://api.github.com/repos/simonw/datasette/issues/1036 MDEyOklzc3VlQ29tbWVudDcxMzE4NjE4OQ== simonw 9599 2020-10-20T22:56:33Z 2020-10-20T22:56:33Z OWNER

I think this plus the binary-CSV stuff in #1034 will justify a dedicated section of the documentation to talk about how Datasette handles binary BLOB columns.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Make it possible to download BLOB data from the Datasette UI 725996507  
713185871 https://github.com/simonw/datasette/issues/1036#issuecomment-713185871 https://api.github.com/repos/simonw/datasette/issues/1036 MDEyOklzc3VlQ29tbWVudDcxMzE4NTg3MQ== simonw 9599 2020-10-20T22:55:36Z 2020-10-20T22:55:36Z OWNER

I can also use a Content-Disposition header to force a download. I'm reasonably confident that the combination of Content-Disposition and X-Content-Type-Options: nosniff and application/binary will let me allow users to download the contents of arbitrary BLOB columns without any XSS risk.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Make it possible to download BLOB data from the Datasette UI 725996507  
713185173 https://github.com/simonw/datasette/issues/1036#issuecomment-713185173 https://api.github.com/repos/simonw/datasette/issues/1036 MDEyOklzc3VlQ29tbWVudDcxMzE4NTE3Mw== simonw 9599 2020-10-20T22:53:41Z 2020-10-20T22:53:41Z OWNER

https://security.stackexchange.com/questions/12896/does-x-content-type-options-really-prevent-content-sniffing-attacks says:

In Tangled Web Michal Zalewski says:

Refrain from using Content-Type: application/octet-stream and use application/binary instead, especially for unknown document types. Refrain from returning Content-Type: text/plain.

For example, any code-hosting platform must exercise caution when returning executables or source archives as application/octet-stream, because there is a risk they may be misinterpreted as HTML and displayed inline.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Make it possible to download BLOB data from the Datasette UI 725996507  
713184374 https://github.com/simonw/datasette/issues/1036#issuecomment-713184374 https://api.github.com/repos/simonw/datasette/issues/1036 MDEyOklzc3VlQ29tbWVudDcxMzE4NDM3NA== simonw 9599 2020-10-20T22:51:22Z 2020-10-20T22:51:22Z OWNER

From https://hackerone.com/reports/126197:

archive.uber.com mirrors pypi. When downloading .tar.gz files from archive.uber.com, the MIME type is application/octet-stream. Injecting <html><script>alert(0)</script> into the start of the .tar.gz causes an XSS in Internet Explorer due to MIME sniffing.

So you do have to be careful not to open accidental XSS holes with application/octet-stream thanks to (presumably older) versions of IE.

From that thread it looks like the solution is to add a X-Content-Type-Options: nosniff header.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Make it possible to download BLOB data from the Datasette UI 725996507  
713183306 https://github.com/simonw/datasette/issues/1036#issuecomment-713183306 https://api.github.com/repos/simonw/datasette/issues/1036 MDEyOklzc3VlQ29tbWVudDcxMzE4MzMwNg== simonw 9599 2020-10-20T22:48:10Z 2020-10-20T22:48:10Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Make it possible to download BLOB data from the Datasette UI 725996507  
713176082 https://github.com/simonw/datasette/issues/1034#issuecomment-713176082 https://api.github.com/repos/simonw/datasette/issues/1034 MDEyOklzc3VlQ29tbWVudDcxMzE3NjA4Mg== simonw 9599 2020-10-20T22:27:33Z 2020-10-20T22:27:33Z OWNER

This feels good to me - it's consistent with how other features in Datasette work, and it means users who need the binary data in CSV (for whatever reason) can get it if they want to.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Better way of representing binary data in .csv output 725184645  
713175741 https://github.com/simonw/datasette/issues/1034#issuecomment-713175741 https://api.github.com/repos/simonw/datasette/issues/1034 MDEyOklzc3VlQ29tbWVudDcxMzE3NTc0MQ== simonw 9599 2020-10-20T22:26:45Z 2020-10-20T22:26:45Z OWNER

New idea: since binary in CSV doesn't make sense anyway, emulate Datasette's HTML UI default and output this:

id,title,data
1,Some title,<Binary data: 14 bytes>
2,Other title,<Binary data: 57 bytes>

Then allow users to add ?_base64=1 to the URL to get base64 instead https://twitter.com/simonw/status/1318679950635888641

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Better way of representing binary data in .csv output 725184645  
713174690 https://github.com/simonw/datasette/issues/1034#issuecomment-713174690 https://api.github.com/repos/simonw/datasette/issues/1034 MDEyOklzc3VlQ29tbWVudDcxMzE3NDY5MA== simonw 9599 2020-10-20T22:23:50Z 2020-10-20T22:23:50Z OWNER

Or... default to <Binary data: 7 bytes> and support a ?_base64=1 option which outputs in base64 instead.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Better way of representing binary data in .csv output 725184645  
713174341 https://github.com/simonw/datasette/issues/1034#issuecomment-713174341 https://api.github.com/repos/simonw/datasette/issues/1034 MDEyOklzc3VlQ29tbWVudDcxMzE3NDM0MQ== simonw 9599 2020-10-20T22:22:53Z 2020-10-20T22:23:14Z OWNER

An even easier option: do what the Datasette UI does and output <Binary data: 7 bytes> for that CSV cell, as seen on https://latest.datasette.io/fixtures/binary_data

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Better way of representing binary data in .csv output 725184645  
713172901 https://github.com/simonw/datasette/issues/1034#issuecomment-713172901 https://api.github.com/repos/simonw/datasette/issues/1034 MDEyOklzc3VlQ29tbWVudDcxMzE3MjkwMQ== simonw 9599 2020-10-20T22:19:10Z 2020-10-20T22:20:28Z OWNER

I could go with the same format as datasette-render-binary but using 0x00 as the format for the hex bytes.

0x15 0x1C 0x02 0xC7 JFIF 0x00 0x01

Problem with this is that it's ambiguous: if the ASCII characters 0x15 occur in the text they will be indistinguishable from those hex bytes.

But since representing binary data in CSV fundamentally doesn't make sense I'm not sure if that really matters.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Better way of representing binary data in .csv output 725184645  
713171742 https://github.com/simonw/datasette/issues/741#issuecomment-713171742 https://api.github.com/repos/simonw/datasette/issues/741 MDEyOklzc3VlQ29tbWVudDcxMzE3MTc0Mg== simonw 9599 2020-10-20T22:16:25Z 2020-10-20T22:16:25Z OWNER

See also #992 which will rename --config to --setting.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Replace "datasette publish --extra-options" with "--setting" 607223136  
713170979 https://github.com/simonw/datasette/issues/262#issuecomment-713170979 https://api.github.com/repos/simonw/datasette/issues/262 MDEyOklzc3VlQ29tbWVudDcxMzE3MDk3OQ== simonw 9599 2020-10-20T22:14:37Z 2020-10-20T22:14:37Z OWNER

I think it's worth having a plugin hook for this - it can be same hook that is used internally. Maybe register_extra - it lets you return one or more extra implementations, each with a name and an async function that gets called.

Things like suggested facets will become register_extra hooks. Maybe actual facets too?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add ?_extra= mechanism for requesting extra properties in JSON 323658641  
713170284 https://github.com/simonw/datasette/issues/262#issuecomment-713170284 https://api.github.com/repos/simonw/datasette/issues/262 MDEyOklzc3VlQ29tbWVudDcxMzE3MDI4NA== simonw 9599 2020-10-20T22:13:01Z 2020-10-20T22:13:01Z OWNER

In the documentation for ?_extra= I think I'll emphasize the comma-separated version of it. Also: there will be ?_extra= values which act as aliases for collection combinations - e.g. ?_extra=full will toggle everything.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add ?_extra= mechanism for requesting extra properties in JSON 323658641  
712986115 https://github.com/simonw/datasette/issues/782#issuecomment-712986115 https://api.github.com/repos/simonw/datasette/issues/782 MDEyOklzc3VlQ29tbWVudDcxMjk4NjExNQ== simonw 9599 2020-10-20T16:28:46Z 2020-10-20T16:29:51Z OWNER

I think this all comes down to how the ?_extras= mechanism works (see #262), as first hinted at in a30c5b220c15360d575e94b0e67f3255e120b916 (see commit message) when I added this long-forgotten undocumented feature: https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables

Extras need to be able to execute additional SQL, since that would solve the problem we have now where the expensive "suggested facets" code runs on all .json output even when its results are not being shown.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Redesign default .json format 627794879  
712976314 https://github.com/simonw/datasette/issues/1035#issuecomment-712976314 https://api.github.com/repos/simonw/datasette/issues/1035 MDEyOklzc3VlQ29tbWVudDcxMjk3NjMxNA== simonw 9599 2020-10-20T16:21:42Z 2020-10-20T16:21:42Z OWNER

Makes me question if datasette.urls should grow functionality equivalent to the other path and querystring manipulation methods in datasette.utils:

https://github.com/simonw/datasette/blob/66120a7a1cb592e8a21164cf537f62a4d7ab1dfc/datasette/utils/init.py#L216-L279

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette.urls.table(..., format="json") argument 725743755  
712965574 https://github.com/simonw/datasette/issues/1035#issuecomment-712965574 https://api.github.com/repos/simonw/datasette/issues/1035 MDEyOklzc3VlQ29tbWVudDcxMjk2NTU3NA== simonw 9599 2020-10-20T16:13:57Z 2020-10-20T16:13:57Z OWNER

That renderers[key] = path_with_format( is in a base class which can be used for both arbitrary queries, canned queries and the table view. I think that's OK, but it means that the format="json" argument on datasette.urls.table() won't be used by Datasette internally, it will just be available for plugins.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette.urls.table(..., format="json") argument 725743755  
712963959 https://github.com/simonw/datasette/issues/1035#issuecomment-712963959 https://api.github.com/repos/simonw/datasette/issues/1035 MDEyOklzc3VlQ29tbWVudDcxMjk2Mzk1OQ== simonw 9599 2020-10-20T16:11:25Z 2020-10-20T16:11:25Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette.urls.table(..., format="json") argument 725743755  
712962517 https://github.com/simonw/datasette/issues/1026#issuecomment-712962517 https://api.github.com/repos/simonw/datasette/issues/1026 MDEyOklzc3VlQ29tbWVudDcxMjk2MjUxNw== simonw 9599 2020-10-20T16:09:12Z 2020-10-20T16:09:12Z OWNER

That datasette.urls.table("db", "table") + ".json" example is bad because if the table name contains a . it should be ?_format=json instead.

Maybe .table() should have a format="json" option that knows how to do this.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 How should datasette.client interact with base_url 722738988  
712959034 https://github.com/simonw/datasette/issues/1026#issuecomment-712959034 https://api.github.com/repos/simonw/datasette/issues/1026 MDEyOklzc3VlQ29tbWVudDcxMjk1OTAzNA== simonw 9599 2020-10-20T16:03:33Z 2020-10-20T16:03:55Z OWNER

Reconsidering this: I think the .get() etc methods should automatically add the base_url prefix for you, since these APIs are only intended to make internal calls.

The clincher on this is when I went to add a section to the datasette.client documentation recommending you use datasette.urls.path() for every call to them that you make.

But there's a problem: to handle table name escaping users are likely to want to use datasette.urls.table() anyway, like this:

response = await datasette.client.get(datasette.urls.table("db", "table") + ".json")

This risks adding the base_url prefix twice. Maybe the .table() method could return a string-like object that is marked as already having the base_url prefix added, so the client.get() method knows not to add it again.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 How should datasette.client interact with base_url 722738988  
712855389 https://github.com/simonw/datasette/issues/991#issuecomment-712855389 https://api.github.com/repos/simonw/datasette/issues/991 MDEyOklzc3VlQ29tbWVudDcxMjg1NTM4OQ== furilo 24740 2020-10-20T13:36:41Z 2020-10-20T13:36:41Z NONE

Here is one quick sketch (done in Figma :P) for an idea: a possible filter to switch between showing all tables from all databases, or grouping tables by database.

(the switch is interactive)

All tables: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=1%3A2&viewport=536%2C348%2C0.5&scaling=min-zoom

Grouped: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=3%3A974&viewport=536%2C348%2C0.5&scaling=min-zoom

When only 1 database: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=1%3A162&viewport=536%2C348%2C0.5&scaling=min-zoom

Is this is useful, I can send some more suggestions/sketches. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Redesign application homepage 714377268  
712607608 https://github.com/simonw/datasette/issues/1023#issuecomment-712607608 https://api.github.com/repos/simonw/datasette/issues/1023 MDEyOklzc3VlQ29tbWVudDcxMjYwNzYwOA== simonw 9599 2020-10-20T05:47:42Z 2020-10-20T05:47:42Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Fix issues relating to base_url 722673818  
712607227 https://github.com/simonw/datasette/issues/1026#issuecomment-712607227 https://api.github.com/repos/simonw/datasette/issues/1026 MDEyOklzc3VlQ29tbWVudDcxMjYwNzIyNw== simonw 9599 2020-10-20T05:46:44Z 2020-10-20T05:46:44Z OWNER

We have a solution for this now: datasette.urls from #1033 can be used by plugins to assemble the correct URLs to pass to .get() and friends.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 How should datasette.client interact with base_url 722738988  
712604541 https://github.com/simonw/datasette/issues/1023#issuecomment-712604541 https://api.github.com/repos/simonw/datasette/issues/1023 MDEyOklzc3VlQ29tbWVudDcxMjYwNDU0MQ== simonw 9599 2020-10-20T05:39:44Z 2020-10-20T05:39:44Z OWNER

Here's the alpha with most of this work ready for people to preview: https://github.com/simonw/datasette/releases/tag/0.51a0

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Fix issues relating to base_url 722673818  
712604364 https://github.com/simonw/datasette/issues/838#issuecomment-712604364 https://api.github.com/repos/simonw/datasette/issues/838 MDEyOklzc3VlQ29tbWVudDcxMjYwNDM2NA== simonw 9599 2020-10-20T05:39:15Z 2020-10-20T05:39:15Z OWNER

OK, I've made a ton of improvements to how the base_url setting works - see tickets linked from #1023. I've just pushed out an alpha release with those changes in it: https://github.com/simonw/datasette/releases/tag/0.51a0

@tsibley @tballison @ChristopherWilks I'd really appreciate your help testing this alpha!

You can install it with:

pip install datasette==0.51a0

It should work with just ProxyPass, without needing the ProxyPassReverse setting.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Incorrect URLs when served behind a proxy with base_url set 637395097  
712597762 https://github.com/simonw/datasette/issues/865#issuecomment-712597762 https://api.github.com/repos/simonw/datasette/issues/865 MDEyOklzc3VlQ29tbWVudDcxMjU5Nzc2Mg== simonw 9599 2020-10-20T05:22:59Z 2020-10-20T05:22:59Z OWNER

OK, this is definitely working now.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 base_url doesn't seem to work when adding criteria and clicking "apply" 644582921  
712593790 https://github.com/simonw/datasette/issues/1025#issuecomment-712593790 https://api.github.com/repos/simonw/datasette/issues/1025 MDEyOklzc3VlQ29tbWVudDcxMjU5Mzc5MA== simonw 9599 2020-10-20T05:12:36Z 2020-10-20T05:12:36Z OWNER

I'm going to leave the cookies code setting cookies to default to the "/" top level.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Fix last remaining links to "/" that do not respect base_url 722724086  
712590398 https://github.com/simonw/datasette/issues/782#issuecomment-712590398 https://api.github.com/repos/simonw/datasette/issues/782 MDEyOklzc3VlQ29tbWVudDcxMjU5MDM5OA== simonw 9599 2020-10-20T05:03:46Z 2020-10-20T05:04:09Z OWNER

OK, https://latest-with-plugins.datasette.io/ is running that now - e.g. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions.json-preview or https://latest-with-plugins.datasette.io/fixtures/compound_three_primary_keys.json-preview

json { "rows": [ { "pk": 1, "name": "The Mystery Spot", "address": "465 Mystery Spot Road, Santa Cruz, CA 95065", "latitude": 37.0167, "longitude": -122.0024 }, { "pk": 2, "name": "Winchester Mystery House", "address": "525 South Winchester Boulevard, San Jose, CA 95128", "latitude": 37.3184, "longitude": -121.9511 }, { "pk": 3, "name": "Burlingame Museum of PEZ Memorabilia", "address": "214 California Drive, Burlingame, CA 94010", "latitude": 37.5793, "longitude": -122.3442 }, { "pk": 4, "name": "Bigfoot Discovery Museum", "address": "5497 Highway 9, Felton, CA 95018", "latitude": 37.0414, "longitude": -122.0725 } ], "total": 4, "next_url": null }

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Redesign default .json format 627794879  
712585921 https://github.com/simonw/datasette/issues/782#issuecomment-712585921 https://api.github.com/repos/simonw/datasette/issues/782 MDEyOklzc3VlQ29tbWVudDcxMjU4NTkyMQ== simonw 9599 2020-10-20T04:48:01Z 2020-10-20T04:48:01Z OWNER

I'll update datasette-json-preview with that now.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Redesign default .json format 627794879  
712585687 https://github.com/simonw/datasette/issues/782#issuecomment-712585687 https://api.github.com/repos/simonw/datasette/issues/782 MDEyOklzc3VlQ29tbWVudDcxMjU4NTY4Nw== simonw 9599 2020-10-20T04:47:02Z 2020-10-20T04:47:12Z OWNER

Great point about CORS, I hadn't considered that.

I think I'm going to keep the Link: header (added in #1014) because I quite enjoy using it with GitHub and WordPress, but I'm not going to have it be the default way of doing pagination. For the default shape I'm now leaning towards this:

json { "total": 36, "rows": [{"id": 1, "name": "Cleo"}], "next_url": "https://latest-with-plugins.datasette.io/fixtures/facetable.json?_next=5" }

So three keys: total, rows and next_url. Then extra keys can be added using ?_extra= with various named bundles.

 
{
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Redesign default .json format 627794879  
712582699 https://github.com/simonw/datasette/issues/1034#issuecomment-712582699 https://api.github.com/repos/simonw/datasette/issues/1034 MDEyOklzc3VlQ29tbWVudDcxMjU4MjY5OQ== simonw 9599 2020-10-20T04:36:04Z 2020-10-20T04:36:14Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Better way of representing binary data in .csv output 725184645  
712581994 https://github.com/simonw/datasette/issues/1034#issuecomment-712581994 https://api.github.com/repos/simonw/datasette/issues/1034 MDEyOklzc3VlQ29tbWVudDcxMjU4MTk5NA== simonw 9599 2020-10-20T04:33:28Z 2020-10-20T04:33:28Z OWNER

The datasette-render-binary plugin does this, which I really like - but without the different coloured fonts I'm not sure how readable it would be as just plain text:

Really the goal here is to find the most human-friendly option, so that people looking at the output have a vague idea what's going on. That's why I'm not leaping at the chance to use base64.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Better way of representing binary data in .csv output 725184645  
712580976 https://github.com/simonw/datasette/issues/1034#issuecomment-712580976 https://api.github.com/repos/simonw/datasette/issues/1034 MDEyOklzc3VlQ29tbWVudDcxMjU4MDk3Ng== simonw 9599 2020-10-20T04:29:23Z 2020-10-20T04:29:23Z OWNER

Most obvious option is base64. Any other potential solutions I'm missing?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Better way of representing binary data in .csv output 725184645  
712579674 https://github.com/simonw/datasette/issues/1025#issuecomment-712579674 https://api.github.com/repos/simonw/datasette/issues/1025 MDEyOklzc3VlQ29tbWVudDcxMjU3OTY3NA== simonw 9599 2020-10-20T04:24:10Z 2020-10-20T04:24:10Z OWNER

Changed my mind, error.html needs access to urls in order to link to its CSS file. Passing it after all (it already got passed ds.config("base_url") so ds was available previously).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Fix last remaining links to "/" that do not respect base_url 722724086  
712569695 https://github.com/simonw/datasette/issues/782#issuecomment-712569695 https://api.github.com/repos/simonw/datasette/issues/782 MDEyOklzc3VlQ29tbWVudDcxMjU2OTY5NQ== carlmjohnson 222245 2020-10-20T03:45:48Z 2020-10-20T03:46:14Z NONE

I vote against headers. It has a lot of strikes against it: poor discoverability, new developers often don’t know how to use them, makes CORS harder, makes it hard to use eg with JQ, needs ad hoc specification for each bit of metadata, etc.

The only advantage of headers is that you don’t need to do .rows, but that’s actually good as a data validation step anyway—if .rows is missing assume there’s an error and do your error handling path instead of parsing the rest.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Redesign default .json format 627794879  
712529413 https://github.com/simonw/datasette/issues/1033#issuecomment-712529413 https://api.github.com/repos/simonw/datasette/issues/1033 MDEyOklzc3VlQ29tbWVudDcxMjUyOTQxMw== simonw 9599 2020-10-20T01:21:12Z 2020-10-20T01:21:12Z OWNER

Also refs #1023

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette.urls.static_plugins(...) method 725099777  
712525557 https://github.com/simonw/datasette/issues/1025#issuecomment-712525557 https://api.github.com/repos/simonw/datasette/issues/1025 MDEyOklzc3VlQ29tbWVudDcxMjUyNTU1Nw== simonw 9599 2020-10-20T01:07:02Z 2020-10-20T01:07:02Z OWNER

I fixed the queries.html one. I'm not going to fix these two: datasette/templates/error.html: <a href="/">home</a> datasette/templates/patterns.html: <a href="/">home</a> / Because the error.html one does not get passed a context (which makes sense since an error has occurred) and the pattern portfolio doesn't need to link to anywhere in particular.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Fix last remaining links to "/" that do not respect base_url 722724086  
712524699 https://github.com/simonw/datasette/issues/904#issuecomment-712524699 https://api.github.com/repos/simonw/datasette/issues/904 MDEyOklzc3VlQ29tbWVudDcxMjUyNDY5OQ== simonw 9599 2020-10-20T01:04:12Z 2020-10-20T01:04:12Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates 663228985  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);