{"html_url": "https://github.com/simonw/datasette/issues/1024#issuecomment-709625063", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1024", "id": 709625063, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTYyNTA2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-15T22:33:22Z", "updated_at": "2020-10-15T22:33:22Z", "author_association": "OWNER", "body": "Of those errors...\r\n\r\n`http://localhost:8000/robots.txt` 404 is fine.\r\n\r\n`http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22` looks to me like a `wget` parsing bug where it got confused by this JavaScript:\r\n\r\n```\r\nwindow.DATASETTE_CLUSTER_MAP_TILE_LAYER_OPTIONS = {\"maxZoom\": 19, \"detectRetina\": true, \"attribution\": \"© OpenStreetMap contributors\"};\r\n```\r\n\r\n`http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js` is a real bug. It's a bug in `datasette-cluster-map` but also requires me to solve #988 - mechanism for plugins to construct URLs that obey `base_url`.\r\n\r\nI'm not sure why I'm getting a hit to `http://localhost:8000/` since I wouldn't expect to link to `/` anywhere.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722674708, "label": "Figure out how to run an environment that exercises the base_url proxy setting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1024#issuecomment-709622973", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1024", "id": 709622973, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTYyMjk3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-15T22:27:31Z", "updated_at": "2020-10-15T22:27:31Z", "author_association": "OWNER", "body": "Here's how I tested it:\r\n```\r\ntime wget -r 'http://localhost:8000/datasette/' 2>&1 | grep -i -C 5 \"failed\\|error\" > /tmp/errors.txt\r\n```\r\nThis wrote out any errors (plus context) to the `errors.txt` log - and reported that the full crawl took 33s.\r\n\r\nHere's what I got in `errors.txt`:\r\n```\r\n\r\n 0K . 71.6M=0s\r\n\r\n2020-10-15 15:23:09 (71.6 MB/s) - \u2018localhost:8000/datasette/index.html\u2019 saved [1276]\r\n\r\nLoading robots.txt; please ignore errors.\r\n--2020-10-15 15:23:09-- http://localhost:8000/robots.txt\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 404 Not Found\r\n--\r\n--2020-10-15 15:23:09-- http://localhost:8000/robots.txt\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 404 Not Found\r\n2020-10-15 15:23:09 ERROR 404: Not Found.\r\n\r\n--2020-10-15 15:23:09-- http://localhost:8000/datasette/-/static/app.css?b576be\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 8563 (8.4K) [text/css]\r\n--\r\n--\r\n2020-10-15 15:23:13 (7.90 MB/s) - \u2018localhost:8000/datasette/fixtures/primary_key_multiple_columns_explicit_label.json?_shape=object\u2019 saved [58]\r\n\r\n--2020-10-15 15:23:13-- http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 404 Not Found\r\n2020-10-15 15:23:13 ERROR 404: Not Found.\r\n\r\n--2020-10-15 15:23:13-- http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+limit+101\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: unspecified [text/html]\r\n--\r\n--\r\n2020-10-15 15:23:13 (84.3 MB/s) - \u2018localhost:8000/datasette/fixtures/roadside_attractions.json?_shape=object\u2019 saved [619]\r\n\r\n--2020-10-15 15:23:13-- http://localhost:8000/datasette/fixtures/%5C%22https://www.openstreetmap.org/copyright%5C%22\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 404 Not Found\r\n2020-10-15 15:23:13 ERROR 404: Not Found.\r\n\r\n--2020-10-15 15:23:13-- http://localhost:8000/datasette/fixtures?sql=select+pk%2C+text1%2C+text2%2C+%5Bname+with+.+and+spaces%5D+from+searchable+order+by+pk+limit+101\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: unspecified [text/html]\r\n--\r\n--\r\n2020-10-15 15:23:14 (28.6 MB/s) - \u2018localhost:8000/datasette/fixtures/searchable_view_configured_by_metadata.json?_shape=array&_nl=on\u2019 saved [180]\r\n\r\n--2020-10-15 15:23:14-- http://localhost:8000/\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 404 Not Found\r\n2020-10-15 15:23:14 ERROR 404: Not Found.\r\n\r\n--2020-10-15 15:23:14-- http://localhost:8000/datasette/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101&_hide_sql=1\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: unspecified [text/html]\r\n--\r\n--\r\n2020-10-15 15:23:21 (64.1 MB/s) - \u2018localhost:8000/datasette/fixtures.csv?sql=select+pk,+name,+address,+latitude,+longitude+from+roadside_attractions+order+by+pk+limit+101&_size=max\u2019 saved [403]\r\n\r\n--2020-10-15 15:23:21-- http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 404 Not Found\r\n2020-10-15 15:23:21 ERROR 404: Not Found.\r\n\r\n--2020-10-15 15:23:21-- http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+desc+limit+101\r\nReusing existing connection to localhost:8000.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: unspecified [text/html]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722674708, "label": "Figure out how to run an environment that exercises the base_url proxy setting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1024#issuecomment-709600335", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1024", "id": 709600335, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTYwMDMzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-15T21:28:02Z", "updated_at": "2020-10-15T22:25:43Z", "author_association": "OWNER", "body": "This is working OK so far:\r\n\r\n\"fixtures__compound_three_primary_keys__1_001_rows_where_sorted_by_pk2\"\r\n\r\nI'll try crawling it with `wget -r` to see if I get any errors.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722674708, "label": "Figure out how to run an environment that exercises the base_url proxy setting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1024#issuecomment-709598324", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1024", "id": 709598324, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTU5ODMyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-15T21:23:33Z", "updated_at": "2020-10-15T21:26:55Z", "author_association": "OWNER", "body": "Combining these two examples, here's the config file I am going to use for this. I'll save this as `nginx.conf`:\r\n```\r\ndaemon off;\r\n\r\nevents {\r\n worker_connections 1024;\r\n}\r\n\r\nhttp {\r\n server {\r\n listen 8000;\r\n\r\n location /datasette {\r\n proxy_pass http://127.0.0.1:8001;\r\n proxy_set_header X-Real-IP $remote_addr;\r\n proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\r\n }\r\n }\r\n}\r\n```\r\nThen start the server with:\r\n```\r\nnginx -p `pwd` -c `pwd`/nginx.conf\r\n```\r\nAnd start Datasette like this:\r\n```\r\ndatasette fixtures.db --config base_url:/datasette/\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722674708, "label": "Figure out how to run an environment that exercises the base_url proxy setting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1024#issuecomment-709597589", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1024", "id": 709597589, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTU5NzU4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-15T21:21:53Z", "updated_at": "2020-10-15T21:23:25Z", "author_association": "OWNER", "body": "Here's a recipe for running nginx against a custom config file: https://gist.github.com/simonw/35f0ebf9c1d6df158759\r\n\r\n```\r\ndaemon off;\r\n\r\nevents {\r\n worker_connections 1024;\r\n}\r\n\r\nhttp {\r\n access_log /dev/stdout;\r\n error_log /dev/stderr;\r\n\r\n types {\r\n text/html html htm shtml;\r\n text/css css;\r\n image/gif gif;\r\n image/jpeg jpeg jpg;\r\n application/javascript js;\r\n }\r\n\r\n server {\r\n listen 8002;\r\n index index.html;\r\n root app;\r\n }\r\n}\r\n```\r\n```\r\nnginx -p `pwd` -c `pwd`/nginx.conf\r\n```\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722674708, "label": "Figure out how to run an environment that exercises the base_url proxy setting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1024#issuecomment-709595960", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1024", "id": 709595960, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTU5NTk2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-15T21:18:14Z", "updated_at": "2020-10-15T21:18:14Z", "author_association": "OWNER", "body": "Typing `nginx` starts it running as a daemon listening on port `http-alt` aka 8080. It uses the config file from ` /usr/local/etc/nginx/nginx.conf`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722674708, "label": "Figure out how to run an environment that exercises the base_url proxy setting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1024#issuecomment-709590941", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1024", "id": 709590941, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTU5MDk0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-15T21:07:47Z", "updated_at": "2020-10-15T21:07:47Z", "author_association": "OWNER", "body": "On macOS I ran `brew install nginx`. I'm going to try running it on port 8000 so I don't have to run it as root.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722674708, "label": "Figure out how to run an environment that exercises the base_url proxy setting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1024#issuecomment-709590337", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1024", "id": 709590337, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTU5MDMzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-15T21:06:24Z", "updated_at": "2020-10-15T21:07:19Z", "author_association": "OWNER", "body": "From https://stackoverflow.com/questions/32549684/nginx-proxy-and-remove-proxy-pass-prefix/32550251 it looks like the config I should use is:\r\n```\r\nserver {\r\n listen 80;\r\n server_name example.com;\r\n\r\n location /datasette/ {\r\n proxy_pass http://127.0.0.1:8001;\r\n proxy_set_header X-Real-IP $remote_addr;\r\n proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\r\n proxy_set_header X-Forwarded-Proto $scheme;\r\n proxy_read_timeout 90;\r\n }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722674708, "label": "Figure out how to run an environment that exercises the base_url proxy setting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1024#issuecomment-709589297", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1024", "id": 709589297, "node_id": "MDEyOklzc3VlQ29tbWVudDcwOTU4OTI5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-15T21:04:31Z", "updated_at": "2020-10-15T21:04:31Z", "author_association": "OWNER", "body": "I think nginx or Apache would be the best tools for this. I'm inclined to try with nginx first since I know it better.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 722674708, "label": "Figure out how to run an environment that exercises the base_url proxy setting"}, "performed_via_github_app": null}