issue_comments

I ran pytest --pdb -x to drop into the debugger on the first failing test.

``` assert expected_rows == len(fetched)

  assert expected_pages == count

E assert 9 == 5 `` That's intest_paginate_tables_and_views()`.

On a hunch, I checked the current settings:

(Pdb) ds_client.ds._settings {'default_page_size': 50, 'max_returned_rows': 100, 'max_insert_rows': 100, 'num_sql_threads': 1, 'sql_time_limit_ms': 200, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'allow_facet': True, 'allow_download': True, 'allow_signed_tokens': True, 'max_signed_tokens_ttl': 0, 'suggest_facets': True, 'default_cache_ttl': 5, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'trace_debug': False, 'base_url': '/'} Looks like something changed default_page_size to 50 and forgot to change it back!

Also need an alternative mechanism to this convenience for getting CSRF tokens before a POST:

https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/datasette/utils/testing.py#L90-L103

One option would be adding that mechanism to datasette.client.post(...) - as a _csrf_from= parameter (with an underscore because it's mainly intended for use in tests, though perhaps that's a weird convention that I shouldn't introduce).

Looks like that @pytest.mark.ds_client mark I've been using isn't necessary - I added that so I could easily run pytest -m ds_client to execute all tests that I had ported to the new feature, but actually this achieves the same thing:

pytest -k ds_client

So I'm going to remove the mark.

Bad news: they're definitely caused by tests that are subtly affected by other tests.

This passes without errors:

pytest -k test_paginate_tables_and_views

Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI:

======== short test summary info ======== FAILED tests/test_table_api.py::test_paginate_tables_and_views[/fixtures/paginated_view.json-201-9] - assert 9 == 5 FAILED tests/test_table_api.py::test_sortable_columns_metadata - KeyError: 'error' FAILED tests/test_table_api.py::test_searchable_views[/fixtures/searchable_view_configured_by_metadata.json?_search=weasel-expected_rows0] - AssertionError: assert [[2, 'terry d...sel', 'puma']] == [[1, 'barry c...sel', 'puma']] FAILED tests/test_table_api.py::test_unit_filters - KeyError: 'distance' FAILED tests/test_table_api.py::test_custom_query_with_unicode_characters - AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...] FAILED tests/test_table_html.py::test_table_html_no_primary_key - AttributeError: 'NoneType' object has no attribute 'strip' FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['<td class=...1</em></td>']] == [['<td class=...>1</a></td>']] FAILED tests/test_table_html.py::test_view_html - assert <a href="/fixtures/simple_view?_size=3&_sort=upper_content" rel="nofollow">upper_content</a> is None FAILED tests/test_table_html.py::test_table_metadata - AssertionError: assert 'This <em&...t; is escaped' == 'simple_primary_key' FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id\xa0▼', 'name'] FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk\xa0▼', 'name'] FAILED tests/test_table_html.py::test_column_metadata - AttributeError: 'NoneType' object has no attribute 'findAll' ======== 13 failed, 1279 passed, 3 skipped, 57 warnings in 572.40s (0:09:32) ========

Trying this instead: docker run -it ubuntu:22.04 /bin/bash Then in that shell: apt-get update apt-get install python3.11 python3.11-venv git -y cd /tmp git clone https://github.com/simonw/datasette cd datasette git checkout async-tests python3.11 -m venv venv pip install -e '.[test]' pytest

This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop.

https://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though.

https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/

Many of the failing tests pass on my laptop but fail in CI.

I tried adding invoke_startup() to the ds_client() fixture to see if that would fix this.

It did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884

====== 11 failed, 1252 passed, 1 skipped, 1 warning in 185.77s (0:03:05) =======

Opened a follow-up issue here: - #1962

Opened a follow-up issue here: - #1962

Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626 ```

  assert response.json() == [{"id": 1, "name": "San Francisco"}]

E AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] E Full diff: E - [{'id': 1, 'name': 'San Francisco'}] E + {'error': 'Table not found: 𝐜𝐢𝐭𝐢𝐞𝐬', 'ok': False, 'status': 404, 'title': None} `` A hunch: this failure suggests that maybe the fixtures tables were not correctly created when this test run. Maybe that can happen whenpython -n auto` runs a bunch of separate processes and hence one of the tests randomly gets run in a fresh process and executes before the in-memory fixtures database has been fully populated.

I tried adding invoke_startup() to the ds_client() fixture to see if that would fix this.

That's enough for this round. I'll get the tests passing and land this.

This is quite fast: % pytest -m ds_client -n auto ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict gw0 [291] / gw1 [291] / gw2 [291] / gw3 [291] / gw4 [291] / gw5 [291] ................................................................................................................................................................................. [ 60%] .................................................................................................................. [100%] ================================================================================== 291 passed in 6.30s ==================================================================================

Got that done to: 68 passed in 14.92s

By implementing my own global variable - since pytest won't use a global fixture for me, I decided to do it for myself.

It did NOT speed it up:

68 passed in 26.26s

I'm going to do test_table_html.py next.

Currently: 68 passed in 17.20s

Will this speed it up?

The thing with Datasette(memory=False) is tripping me up.

The problem is that the tests written against app_client - which I want to replace - all assume that there is no _memory database, because when you start Datasette with at least one database file it doesn't enable _memory unless you explicitly tell it to.

But the new ds_client fixture works by creating a named in-memory database called fixtures, which it does with a call to ds.add_memory_database("fixtures") after the object has been instantiated.

This results in a datasette instance that DOES have a _memory database, when we didn't want one.

My initial solution attempt was a huge hack - I decided that if you pass memory=False to the Datasette constructor it should mean "don't add a _memory database even though I didn't pass any files". I set a the default memory argument to None.

This is weird and surprising (memory=False no does something different from memory=None?) and I found other tests that it broke, like this one:

python def test_sql_errors_logged_to_stderr(): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, ["--get", "/_memory.json?sql=select+blah"]) assert result.exit_code == 1 assert "sql = 'select blah', params = {}: no such column: blah\n" in result.stderr It ended up with no _memory database because it turns out datasette serve ... passes memory=False without me realizing it.

So I'm going to undo that hack and teach the fixture to do this instead:

python db = ds.add_memory_database("fixtures") ds.remove_database("_memory")

This does seem to help: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 1306c407..af9c7696 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = { }

-@pytest.fixture(scope="session") -def event_loop(): - return asyncio.get_event_loop() - - -@pytest_asyncio.fixture(scope="session") +@pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette from .fixtures import METADATA, PLUGINS_DIR @@ -53,10 +48,11 @@ async def ds_client(): db = ds.add_memory_database("fixtures")

 def prepare(conn):

• conn.executescript(TABLES)
• for sql, params in TABLE_PARAMETERIZED_SQL:
• with conn:
• conn.execute(sql, params)
• if not conn.execute("select count(*) from sqlite_master").fetchone()[0]:
• conn.executescript(TABLES)
• for sql, params in TABLE_PARAMETERIZED_SQL:
• with conn:

• conn.execute(sql, params)

  await db.execute_write_fn(prepare) return ds.client diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py index 4f7bf08c..d588342c 100644 --- a/tests/plugins/my_plugin_2.py +++ b/tests/plugins/my_plugin_2.py @@ -117,7 +117,12 @@ def actor_from_request(datasette, request): def permission_allowed(datasette, actor, action): # Testing asyncio version of permission_allowed async def inner(): - assert 2 == (await datasette.get_database().execute("select 1 + 1")).first()[0] + assert ( + 2 + == ( + await datasette.get_database("_internal").execute("select 1 + 1") + ).first()[0] + ) if action == "this_is_allowed_async": return True elif action == "this_is_denied_async": ``pytest -m ds_client` now passes 134 tests.

Need to get pytest -n auto passing too.

If it does turn out that I can't use scope="session" on this fixture it might not actually be a showstopper: I can take advantage of the fact that memory_name="..." databases stay present in memory for the duration of the process, so I could have it such that each test that uses the ds_client fixture DOES construct a fresh Datasette instance, but doesn't need to populate the database since they can re-use the in-memory database from the previous object.

This is in the 0.63.x branch now, ready to go out in a bug fix release.

To test the fix, I did a fresh checkout of simonw/datasette and edited Dockerfile to look like this: ```dockerfile FROM python:3.11.0-slim-bullseye as build

RUN apt-get update && \ apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg/info/*

RUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \ find /usr/local/lib -name 'pycache' | xargs rm -r && \ rm -rf /root/.cache/pip

EXPOSE 8001 CMD ["datasette"] Then I built it like this: docker build -f Dockerfile -t datasette-pre-click . And ran like this: docker run -p 8001:8001 -v $(pwd):/mnt datasette-pre-click datasette --root -p 8001 -h 0.0.0.0 ``` This exhibited the same problem.

Then I pushed a changed to branch and changed the line to: RUN pip install https://github.com/simonw/datasette/archive/refs/heads/click-echo-root.zip && \ Ran this: docker build -f Dockerfile -t datasette-post-click . And this: docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0 It fixed the problem! datasette % docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0 http://0.0.0.0:8001/-/auth-token?token=6542dcf5c8f34f8d13f4af9ce728359c602469efb54029098562bd06c87ad26d INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)

The fix may just be to switch to click.echo(). https://click.palletsprojects.com/en/8.1.x/api/#click.echo

Confirmed, I just got the same result: ``` - % docker run datasetteproject/datasette pip install datasette-upload-csvs ~ % docker commit $(docker ps -lq) datasette-with-plugins sha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f ~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0 INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)

^CINFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [1] http://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98 `` Note how the auth-token URL is only displayed after you hitCtrl+C`!