html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/issues/1426#issuecomment-902263367,https://api.github.com/repos/simonw/datasette/issues/1426,902263367,IC_kwDOBm6k_c41x3JH,9599,2021-08-19T21:33:51Z,2021-08-19T21:36:28Z,OWNER,"I was worried about if it's possible to allow access to `/fixtures` but deny access to `/fixtures?sql=...` From various answers on Stack Overflow it looks like this should handle that: ``` User-agent: * Disallow: /fixtures? ``` I could use this for tables too - it may well be OK to access table index pages while still avoiding pagination, facets etc. I think this should block both query strings and row pages while allowing the table page itself: ``` User-agent: * Disallow: /fixtures/searchable? Disallow: /fixtures/searchable/* ``` Could even accompany that with a `sitemap.xml` that explicitly lists all of the tables - which would mean adding sitemaps to Datasette core too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136, https://github.com/simonw/datasette/issues/1426#issuecomment-902260338,https://api.github.com/repos/simonw/datasette/issues/1426,902260338,IC_kwDOBm6k_c41x2Zy,9599,2021-08-19T21:28:25Z,2021-08-19T21:29:40Z,OWNER,"Actually it looks like you can send a `sitemap.xml` to Google using an unauthenticated GET request to: https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP According to https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136, https://github.com/simonw/datasette/issues/1426#issuecomment-902260799,https://api.github.com/repos/simonw/datasette/issues/1426,902260799,IC_kwDOBm6k_c41x2g_,9599,2021-08-19T21:29:13Z,2021-08-19T21:29:13Z,OWNER,"Bing's equivalent is: https://www.bing.com/webmasters/help/Sitemaps-3b5cf6ed http://www.bing.com/ping?sitemap=FULL_URL_OF_SITEMAP","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,