issue_comments

Stick a twig in it, this will do for the moment.

I think this test may be to blame:

https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/tests/test_plugins.py#L950-L972

It's over-riding _metadata_local and then failing to set it back to original in a finally: block at the end.

These are all the places that tests touch _metadata_local at the moment: ``` (venv) root@76a81d2417f5:/tmp/datasette/tests# rg _metadata_local
test_facets.py 596: ds._metadata_local = { 605: ds._metadata_local["databases"]["test_facet_size"]["tables"]["neighbourhoods"][

test_permissions.py 62: padlock_client.ds._metadata_local["allow"] = allow 77: del padlock_client.ds._metadata_local["allow"] 522: cascade_app_client.ds._metadata_local = updated_metadata 533: cascade_app_client.ds._metadata_local = previous_metadata 549: previous_metadata = cascade_app_client.ds._metadata_local 551: cascade_app_client.ds._metadata_local = metadata 566: cascade_app_client.ds._metadata_local = previous_metadata 842: perms_ds._metadata_local = updated_metadata 849: perms_ds._metadata_local = previous_metadata

test_html.py 1114: orig = ds_client.ds._metadata_local 1115: ds_client.ds._metadata_local = metadata 1123: ds_client.ds._metadata_local = orig

test_plugins.py 1034: ds_client.ds._metadata_local = { ```

I think I've found the problem. The failing test is this one:

paginated_view.json-201-9

That's this example:

https://github.com/simonw/datasette/blob/cede1efeedbc3d928397d53d5a1611eecc598fde/tests/test_table_api.py#L179-L180

Why is it expected to take 9 pages and not 5, when the definition of that view is this:

sql CREATE VIEW paginated_view AS SELECT content, '- ' || content || ' -' AS content_extra FROM no_primary_key; Because paginated_view has extra configuration in metadata.json:

https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/tests/fixtures.py#L357

So this test is showing that metadata can be used to set an alternative page size for a view.

May have spotted the problem with that test_paginate_tables_and_views test:

(Pdb) path, expected_rows, expected_pages (None, 201, 9) path should not be None here:

https://github.com/simonw/datasette/blob/cede1efeedbc3d928397d53d5a1611eecc598fde/tests/test_table_api.py#L175-L212

No that's not it either - path is reassigned on purpose.

I ran pytest --pdb -x to drop into the debugger on the first failing test.

``` assert expected_rows == len(fetched)

  assert expected_pages == count

E assert 9 == 5 `` That's intest_paginate_tables_and_views()`.

On a hunch, I checked the current settings:

(Pdb) ds_client.ds._settings {'default_page_size': 50, 'max_returned_rows': 100, 'max_insert_rows': 100, 'num_sql_threads': 1, 'sql_time_limit_ms': 200, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'allow_facet': True, 'allow_download': True, 'allow_signed_tokens': True, 'max_signed_tokens_ttl': 0, 'suggest_facets': True, 'default_cache_ttl': 5, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'trace_debug': False, 'base_url': '/'} Looks like something changed default_page_size to 50 and forgot to change it back!

Looks like that @pytest.mark.ds_client mark I've been using isn't necessary - I added that so I could easily run pytest -m ds_client to execute all tests that I had ported to the new feature, but actually this achieves the same thing:

pytest -k ds_client

So I'm going to remove the mark.

Bad news: they're definitely caused by tests that are subtly affected by other tests.

This passes without errors:

pytest -k test_paginate_tables_and_views

Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI:

======== short test summary info ======== FAILED tests/test_table_api.py::test_paginate_tables_and_views[/fixtures/paginated_view.json-201-9] - assert 9 == 5 FAILED tests/test_table_api.py::test_sortable_columns_metadata - KeyError: 'error' FAILED tests/test_table_api.py::test_searchable_views[/fixtures/searchable_view_configured_by_metadata.json?_search=weasel-expected_rows0] - AssertionError: assert [[2, 'terry d...sel', 'puma']] == [[1, 'barry c...sel', 'puma']] FAILED tests/test_table_api.py::test_unit_filters - KeyError: 'distance' FAILED tests/test_table_api.py::test_custom_query_with_unicode_characters - AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...] FAILED tests/test_table_html.py::test_table_html_no_primary_key - AttributeError: 'NoneType' object has no attribute 'strip' FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['<td class=...1</em></td>']] == [['<td class=...>1</a></td>']] FAILED tests/test_table_html.py::test_view_html - assert <a href="/fixtures/simple_view?_size=3&_sort=upper_content" rel="nofollow">upper_content</a> is None FAILED tests/test_table_html.py::test_table_metadata - AssertionError: assert 'This <em&...t; is escaped' == 'simple_primary_key' FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id\xa0▼', 'name'] FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk\xa0▼', 'name'] FAILED tests/test_table_html.py::test_column_metadata - AttributeError: 'NoneType' object has no attribute 'findAll' ======== 13 failed, 1279 passed, 3 skipped, 57 warnings in 572.40s (0:09:32) ========

Trying this instead: docker run -it ubuntu:22.04 /bin/bash Then in that shell: apt-get update apt-get install python3.11 python3.11-venv git -y cd /tmp git clone https://github.com/simonw/datasette cd datasette git checkout async-tests python3.11 -m venv venv pip install -e '.[test]' pytest

This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop.

https://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though.

https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/

Many of the failing tests pass on my laptop but fail in CI.

I tried adding invoke_startup() to the ds_client() fixture to see if that would fix this.

It did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884

====== 11 failed, 1252 passed, 1 skipped, 1 warning in 185.77s (0:03:05) =======

Opened a follow-up issue here: - #1962

Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626 ```

  assert response.json() == [{"id": 1, "name": "San Francisco"}]

E AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] E Full diff: E - [{'id': 1, 'name': 'San Francisco'}] E + {'error': 'Table not found: 𝐜𝐢𝐭𝐢𝐞𝐬', 'ok': False, 'status': 404, 'title': None} `` A hunch: this failure suggests that maybe the fixtures tables were not correctly created when this test run. Maybe that can happen whenpython -n auto` runs a bunch of separate processes and hence one of the tests randomly gets run in a fresh process and executes before the in-memory fixtures database has been fully populated.

I tried adding invoke_startup() to the ds_client() fixture to see if that would fix this.

That's enough for this round. I'll get the tests passing and land this.

This is quite fast: % pytest -m ds_client -n auto ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict gw0 [291] / gw1 [291] / gw2 [291] / gw3 [291] / gw4 [291] / gw5 [291] ................................................................................................................................................................................. [ 60%] .................................................................................................................. [100%] ================================================================================== 291 passed in 6.30s ==================================================================================

Got that done to: 68 passed in 14.92s

By implementing my own global variable - since pytest won't use a global fixture for me, I decided to do it for myself.

It did NOT speed it up:

68 passed in 26.26s

I'm going to do test_table_html.py next.

Currently: 68 passed in 17.20s

Will this speed it up?

The thing with Datasette(memory=False) is tripping me up.

The problem is that the tests written against app_client - which I want to replace - all assume that there is no _memory database, because when you start Datasette with at least one database file it doesn't enable _memory unless you explicitly tell it to.

But the new ds_client fixture works by creating a named in-memory database called fixtures, which it does with a call to ds.add_memory_database("fixtures") after the object has been instantiated.

This results in a datasette instance that DOES have a _memory database, when we didn't want one.

My initial solution attempt was a huge hack - I decided that if you pass memory=False to the Datasette constructor it should mean "don't add a _memory database even though I didn't pass any files". I set a the default memory argument to None.

This is weird and surprising (memory=False no does something different from memory=None?) and I found other tests that it broke, like this one:

python def test_sql_errors_logged_to_stderr(): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, ["--get", "/_memory.json?sql=select+blah"]) assert result.exit_code == 1 assert "sql = 'select blah', params = {}: no such column: blah\n" in result.stderr It ended up with no _memory database because it turns out datasette serve ... passes memory=False without me realizing it.

So I'm going to undo that hack and teach the fixture to do this instead:

python db = ds.add_memory_database("fixtures") ds.remove_database("_memory")

This does seem to help: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 1306c407..af9c7696 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = { }

-@pytest.fixture(scope="session") -def event_loop(): - return asyncio.get_event_loop() - - -@pytest_asyncio.fixture(scope="session") +@pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette from .fixtures import METADATA, PLUGINS_DIR @@ -53,10 +48,11 @@ async def ds_client(): db = ds.add_memory_database("fixtures")

 def prepare(conn):

• conn.executescript(TABLES)
• for sql, params in TABLE_PARAMETERIZED_SQL:
• with conn:
• conn.execute(sql, params)
• if not conn.execute("select count(*) from sqlite_master").fetchone()[0]:
• conn.executescript(TABLES)
• for sql, params in TABLE_PARAMETERIZED_SQL:
• with conn:

• conn.execute(sql, params)

  await db.execute_write_fn(prepare) return ds.client diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py index 4f7bf08c..d588342c 100644 --- a/tests/plugins/my_plugin_2.py +++ b/tests/plugins/my_plugin_2.py @@ -117,7 +117,12 @@ def actor_from_request(datasette, request): def permission_allowed(datasette, actor, action): # Testing asyncio version of permission_allowed async def inner(): - assert 2 == (await datasette.get_database().execute("select 1 + 1")).first()[0] + assert ( + 2 + == ( + await datasette.get_database("_internal").execute("select 1 + 1") + ).first()[0] + ) if action == "this_is_allowed_async": return True elif action == "this_is_denied_async": ``pytest -m ds_client` now passes 134 tests.

Need to get pytest -n auto passing too.

If it does turn out that I can't use scope="session" on this fixture it might not actually be a showstopper: I can take advantage of the fact that memory_name="..." databases stay present in memory for the duration of the process, so I could have it such that each test that uses the ds_client fixture DOES construct a fresh Datasette instance, but doesn't need to populate the database since they can re-use the in-memory database from the previous object.

This is all very broken: ``` % pytest -x --pdb ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items

tests/test_package.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_cli_serve_get.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_black.py . [ 0%] tests/test_api.py E

traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

fixturedef = <FixtureDef argname='event_loop' scope='session' baseid='tests'>, request = <SubRequest 'event_loop' for \<Function test_homepage>>

@pytest.hookimpl(hookwrapper=True)
def pytest_fixture_setup(
    fixturedef: FixtureDef, request: SubRequest
) -> Optional[object]:
    """Adjust the event loop policy when an event loop is produced."""
    if fixturedef.argname == "event_loop":
        outcome = yield

      loop = outcome.get_result()

/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pytest_asyncio/plugin.py:377:

/Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: in event_loop return asyncio.get_event_loop()

self = <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x111e34670>

def get_event_loop(self):
    """Get the event loop for the current context.

    Returns an instance of EventLoop or raises an exception.
    """
    if (self._local._loop is None and
            not self._local._set_called and
            threading.current_thread() is threading.main_thread()):
        self.set_event_loop(self.new_event_loop())

    if self._local._loop is None:

      raise RuntimeError('There is no current event loop in thread %r.'
                           % threading.current_thread().name)

E RuntimeError: There is no current event loop in thread 'MainThread'.

/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py:656: RuntimeError

entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py(656)get_event_loop() -> raise RuntimeError('There is no current event loop in thread %r.' (Pdb) q

=================================================================================== warnings summary ==================================================================================== tests/test_cli.py::test_inspect_cli_writes_to_file tests/test_cli.py::test_inspect_cli /Users/simon/Dropbox/Development/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop loop = asyncio.get_event_loop()

tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error tests/test_cli_serve_get.py::test_serve_with_get /Users/simon/Dropbox/Development/datasette/datasette/cli.py:596: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(ds.invoke_startup())

tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error tests/test_cli_serve_get.py::test_serve_with_get /Users/simon/Dropbox/Development/datasette/datasette/cli.py:599: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(check_databases(ds))

tests/test_api.py::test_homepage /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: DeprecationWarning: There is no current event loop return asyncio.get_event_loop()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================================ short test summary info ================================================================================ ERROR tests/test_api.py::test_homepage - RuntimeError: There is no current event loop in thread 'MainThread'. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Quitting debugger !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ======================================================================== 7 passed, 7 warnings, 1 error in 19.15s ======================================================================== (datasette) datasette % ``` The problem looks to be caused by this:

https://github.com/simonw/datasette/blob/87737aa1ace82fa7b54c60c41471ec9a661f5299/tests/conftest.py#L28-L30

Which I found necessary in order to have async def fixtures that could be shared on the scope="session" basis.

Can I work around this, or is scope="session" for async fixtures incompatible with my test suite for some reason?

I'm going to make .status_code work on TestClient response too, so I don't have to worry about using both status or status_code depending on which kind of object I am using.

I'm going to punt on that for the moment and continue to use app_client for tests that use that mechanism.

Here's an annoying error: ```

  response4 = await ds_client.post(
        "/-/logout",
        csrftoken_from=True,
        cookies={"ds_actor": ds_client.actor_cookie({"id": "test"})},
    )

tests/test_auth.py:88:

self = <datasette.app.DatasetteClient object at 0x1033cc4f0>, path = '/-/logout' kwargs = {'cookies': {'ds_actor': 'eyJhIjp7ImlkIjoidGVzdCJ9fQ.fuFCTJG5XE-RNnUM7dcnXx9sPvE'}, 'csrftoken_from': True}, client = <httpx.AsyncClient object at 0x10347a9b0>

async def post(self, path, **kwargs):
    await self.ds.invoke_startup()
    async with httpx.AsyncClient(app=self.app) as client:

      return await client.post(self._fix(path), **kwargs)

E TypeError: AsyncClient.post() got an unexpected keyword argument 'csrftoken_from' `` I need an alternative to thecsrftoken_frommechanism I built forTestClient`:

https://github.com/simonw/datasette/blob/0b68996cc511b3a801f0cd0157bd66332d75f46f/datasette/utils/testing.py#L77-L103

I'm using this new mark: python @pytest.mark.ds_client Purely so I can run all of the tests that I've refactored using:

pytest -m ds_client I'll likely remove this once the test refactoring project is complete.