I think I can tell what the current file limit is like so: ``` In [1]: import resource

In [2]: resource.getrlimit(resource.RLIMIT_NOFILE)
Out[2]: (256, 9223372036854775807) ``` So maybe I should have Datasette refuse to open more database files than that number minus 5 (to give me some spare room for opening CSS files etc).

I also eventually get this error: Traceback (most recent call last): File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 121, in route_path return await view(new_scope, receive, send) File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 336, in inner_static await asgi_send_file(send, full_path, chunk_size=chunk_size) File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 303, in asgi_send_file async with aiofiles.open(str(filepath), mode="rb") as fp: File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/aiofiles/base.py", line 78, in __aenter__ File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/aiofiles/threadpool/__init__.py", line 35, in _open File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run OSError: [Errno 24] Too many open files: '/Users/simonw/Dropbox/Development/datasette/datasette/static/app.css'

While running it against a nested directory with a TON of databases I kept seeing errors like this: Traceback (most recent call last): File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 121, in route_path return await view(new_scope, receive, send) File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 193, in view request, **scope["url_route"]["kwargs"] File "/Users/simonw/Dropbox/Development/datasette/datasette/views/index.py", line 58, in get tables[table]["num_relationships_for_sorting"] = count KeyError: 'primary-candidates-2018/rep_candidates' And Traceback (most recent call last): File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 121, in route_path return await view(new_scope, receive, send) File "/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py", line 193, in view request, **scope["url_route"]["kwargs"] File "/Users/simonw/Dropbox/Development/datasette/datasette/views/index.py", line 58, in get tables[table]["num_relationships_for_sorting"] = count KeyError: 'space_used'

Here's the new utility function I should be using to verify database files that I find:

https://github.com/simonw/datasette/blob/6aa516d82dea9885cb4db8d56ec2ccfd4cd9b840/datasette/utils/init.py#L773-L787

And for https://github.com/simonw/jupyterserverproxy-datasette-demo setting "absolute_url": True appears to fix the bug as well (it previously caused an infinite redirect loop but that seems not to happen any more with the latest base_url setting): https://github.com/simonw/jupyterserverproxy-datasette-demo/commit/880f9d50566357160b793557f145c99e74ad759a

Hooray! This is a really great fix, thanks.

I think the solution here may be to let the incoming x-original-uri header over-ride the raw_path used by Datasette. This may require a new Datasette setting - maybe this:

--config path_from_header:x-original-uri

Woohoo that worked!

Here's the output of the /-/asgi-scope plugin: {'client': ('10.12.3.15', 0), 'headers': [(b'cookie', b''), (b'upgrade-insecure-requests', b'1'), (b'dnt', b'1'), (b'accept-encoding', b'gzip'), (b'accept-language', b'en-US,en;q=0.5'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'webp,*/*;q=0.8'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:74.0) Gecko' b'/20100101 Firefox/74.0'), (b'x-scheme', b'https'), (b'x-original-uri', b'/user/wragge-datasette-test-czqms2oy/proxy/8001/-/asgi-scope'), (b'x-forwarded-proto', b'https,http'), (b'x-forwarded-port', b'443,80'), (b'x-forwarded-host', b'hub.gke.mybinder.org'), (b'x-forwarded-for', b'148.64.98.14,10.12.3.15'), (b'x-real-ip', b'148.64.98.14'), (b'connection', b'close'), (b'host', b'hub.gke.mybinder.org'), (b'x-forwarded-context', b'/user/wragge-datasette-test-czqms2oy/proxy/8001'), (b'x-proxycontextpath', b'/user/wragge-datasette-test-czqms2oy/proxy/8001')], 'http_version': '1.1', 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', 'root_path': '', 'scheme': 'https,http', 'server': ('127.0.0.1', 8001), 'type': 'http'} From this output... it looks like the path being passed through to Datasette is still /-/asgi-scope here when I though it would be /user/wragge-datasette-test-czqms2oy/proxy/8001/-/asgi-scope

The original URL path (the one I want) is passed in the x-original-uri HTTP header.

I'm running this latest release like so:

Launch this binder: https://mybinder.org/v2/gh/wragge/datasette-test/master?urlpath=lab

Open the notebook.

In the notebook run this:

!pip install datasette-debug-asgi https://github.com/simonw/datasette/archive/7b205af0b53c92d17b69f8c95d70a24a0c06ebce.zip

And use this block to run Datasette:

```

Get current running servers

servers = list_running_servers()

Get the current base url

base_url = next(servers)['base_url']

Create a base url for Datasette suing the proxy path

proxy_url = f'{base_url}proxy/8001/'

Display a link to Datasette

display(HTML(f'

View Datasette (Click on the stop button to close the Datasette server)

Launch Datasette

!datasette -- results.db --port 8001 --config base_url:$proxy_url --config path_from_header:x-original-uri --config template_debug:1 ```

I have a hunch about this: maybe the URLs that work are the ones that use {{ base_url }} directly in the template, while the ones that break are the ones that reconstruct the URL based on the incoming path?

Aha! It turns out my demo at https://github.com/simonw/jupyterserverproxy-datasette-demo suffers from the same bug.

Here's the truncatted HTML for https://hub.gke.mybinder.org/user/wragge-datasette-test-czqms2oy/proxy/8001/results/articles ```html

home / results

Weird... the unit test I just added passes. So suggested facets look like they're OK.

And yet the demo in https://mybinder.org/v2/gh/wragge/datasette-test/master?urlpath=lab doesn't work.