issue_comments

That fixed it.

I went with this to generate the long string: https://github.com/simonw/datasette/blob/0b0c5cd7a94fe3f151a3e10261b5c84ee64f2f18/tests/test_csv.py#L157-L176

I need an alternative way of generating a long string with a shorter URL.

sql select group_concat('abcabcabc', '') from json_each(json_array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)) https://latest.datasette.io/_memory?sql=select+group_concat%28%27abcabcabc%27%2C+%27%27%29+from+json_each%28json_array%281%2C+2%2C+3%2C+4%2C+5%2C+6%2C+7%2C+8%2C+9%2C+10%29%29

Most recent httpx release is 0.24 a couple of weeks ago. Here's what changed in that version: https://github.com/encode/httpx/compare/0.23.3...0.24.0

It looks like that URL limit is new: https://github.com/encode/httpx/commit/57daabf673705954afa94686c0002801c93d31f3#diff-78d8d93b5dd4c77d99c3e2b46b7286ba71a8fd60e92d8bd4eee5fb200b4f87bfR149-R155

```python def urlparse(url: str = "", **kwargs: typing.Optional[str]) -> ParseResult: # Initial basic checks on allowable URLs. # ---------------------------------------

# Hard limit the maximum allowable URL length.
if len(url) > MAX_URL_LENGTH:
    raise InvalidURL("URL too long")

``` https://github.com/encode/httpx/blob/32e25497a36e6222cc64a758c98275b450dac28d/httpx/_urlparse.py#L153-L155

In the debugger: (Pdb) MAX_URL_LENGTH 65536 Weird this only seems to be a problem with httpx on Python 3.7 though.

This is the failing test:

https://github.com/simonw/datasette/blob/249fcf8e3e2a90e763f41b080c1b9ec8017f5005/tests/test_csv.py#L156-L167

In a fresh Datasette checkout I ran:

pipenv shell --python /Users/simon/.pyenv/versions/3.7.16/bin/python

That gave me a virtual environment with 3.7.16 Python.

Then I ran:

pip install -e '.[test]'

Weirdly that gave me a which pytest of /opt/homebrew/bin/pytest which ran the tests on 3.11.

I figured out the location of the virtual environment with which python and then ran this:

``` % /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/bin/pytest tests/test_csv.py ============================================================================================== test session starts =============================================================================================== platform darwin -- Python 3.7.16, pytest-7.3.1, pluggy-1.0.0 SQLite: 3.39.5 rootdir: /private/tmp/datasette configfile: pytest.ini plugins: asyncio-0.21.0, timeout-2.1.0, xdist-3.2.1, anyio-3.6.2 asyncio: mode=strict collected 15 items

tests/test_csv.py ..........F.... [100%]

==================================================================================================== FAILURES ==================================================================================================== ______________ test_max_csv_mb _________________

app_client_csv_max_mb_one = <datasette.utils.testing.TestClient object at 0x107ab5c08>

def test_max_csv_mb(app_client_csv_max_mb_one):
    response = app_client_csv_max_mb_one.get(
        (
            "/fixtures.csv?sql=select+'{}'+"
            "from+compound_three_primary_keys&_stream=1&_size=max"

      ).format("abcdefg" * 10000)
    )

/private/tmp/datasette/tests/test_csv.py:161:

/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:240: in call return call_result.result() /Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:428: in result return self.__get_result() /Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:384: in __get_result raise self._exception /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:306: in main_wrap result = await self.awaitable(args, kwargs) /private/tmp/datasette/datasette/utils/testing.py:76: in get headers=headers, /private/tmp/datasette/datasette/utils/testing.py:167: in _request content=post_body, /private/tmp/datasette/datasette/app.py:1787: in request method, self._fix(path, avoid_path_rewrites), kwargs /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:1528: in request extensions=extensions, /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:346: in build_request url = self._merge_url(url) /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:376: in _merge_url merge_url = URL(url) /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urls.py:113: in init self._uri_reference = urlparse(url, *kwargs)

url = "http://localhost/fixtures.csv?sql=select+'abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcde...gabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg'+from+compound_three_primary_keys&_stream=1&_size=max" kwargs = {}

def urlparse(url: str = "", **kwargs: typing.Optional[str]) -> ParseResult:
    # Initial basic checks on allowable URLs.
    # ---------------------------------------

    # Hard limit the maximum allowable URL length.
    if len(url) > MAX_URL_LENGTH:

      raise InvalidURL("URL too long")

E httpx.InvalidURL: URL too long

/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urlparse.py:155: InvalidURL ============================================================================================ short test summary info ============================================================================================= FAILED tests/test_csv.py::test_max_csv_mb - httpx.InvalidURL: URL too long ========================================================================================== 1 failed, 14 passed in 1.83s ========================================================================================== (datasette) simon@Simons-MacBook-Pro datasette % ```

I don't have 3.7 locally. Trying to install it with pyenv.

brew install pyenv

Then:

pyenv install --list | grep 3.7

Installing:

pyenv install 3.7.16

Output:

Installed Python-3.7.16 to /Users/simon/.pyenv/versions/3.7.16

So the executable is /Users/simon/.pyenv/versions/3.7.16/bin/python.

Failure was on 3.7. I'll try that.

Having trouble replicating this on my laptop. I tried a fresh virtual environment with fresh packages (in case this is a httpx change) but this passed:

pytest tests/test_csv.py