issue_comments: 895500565

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions issue performed_via_github_app
https://github.com/simonw/datasette/issues/1426#issuecomment-895500565 https://api.github.com/repos/simonw/datasette/issues/1426 895500565 IC_kwDOBm6k_c41YEEV 9599 2021-08-09T20:00:04Z 2021-08-09T20:00:04Z OWNER

A few options for how this would work:

  • datasette ... --robots allow
  • datasette ... --setting robots allow

Options could be:

  • allow - allow all crawling
  • deny - deny all crawling
  • limited - allow access to the homepage and the index pages for each database and each table, but disallow crawling any further than that

The "limited" mode is particularly interesting. Could even make it the default, but I think that may be a bit too confusing. Idea would be to get the key pages indexed but use nofollow to discourage crawlers from indexing individual row pages or deep pages like https://datasette.io/content/repos?_facet=owner&_facet=language&_facet_array=topics&topics__arraycontains=sqlite#facet-owner.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 964322136