Codecov Report

Merging #1481 (77542e7) into main (6388617) will decrease coverage by 0.01%. The diff coverage is n/a.

:exclamation: Current head 77542e7 differs from pull request most recent head 50005bd. Consider uploading reports for the commit 50005bd to get more accurate results

```diff @@ Coverage Diff @@

main #1481 +/-

========================================== - Coverage 91.83% 91.82% -0.02%
========================================== Files 34 34
Lines 4421 4426 +5
========================================== + Hits 4060 4064 +4
- Misses 361 362 +1
```

| Impacted Files | Coverage Δ | | |---|---|---| | datasette/app.py | 95.36% <0.00%> (-0.14%) | :arrow_down: | | datasette/views/base.py | 95.41% <0.00%> (ø) | | | datasette/views/database.py | 97.56% <0.00%> (ø) | | | datasette/views/table.py | 96.00% <0.00%> (+<0.01%) | :arrow_up: | | datasette/utils/__init__.py | 94.78% <0.00%> (+0.02%) | :arrow_up: |

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 6388617...50005bd. Read the comment docs.

Since that Janus PR hasn't been merged yet, one temporary option for a fix would be to entirely vendor the fixed Janus - https://github.com/aio-libs/janus/blob/9e13d3fb74e2c93d7501443b370a455d1b302b1f/janus/init.py - since it's only a single module.

This is blocking an upgrade for the Homebrew Datasette package: https://github.com/Homebrew/homebrew-core/pull/86932

I applied my PR against Janus to my local copy of Datasette like so:

pip uninstall janus
pip install https://github.com/aio-libs/janus/archive/9e13d3fb74e2c93d7501443b370a455d1b302b1f.zip

Then I ran the Datasette tests and got a much happier pass rate.

I submitted a PR to Janus with a workaround for this: https://github.com/aio-libs/janus/pull/359

There's a tiny chance this could be a bug in Python 3.10 itself - I filed an issue here: https://bugs.python.org/issue45416 - in which I said:

In Python 3.10 it is not possible to instantiate an asyncio.Condition that wraps an asyncio.Lock without raising a "loop argument must agree with lock" exception.

This code raises that exception:

asyncio.Condition(asyncio.Lock())

This worked in previous Python versions.

Note that the error only occurs if an event loop is running. Here's a simple script that replicates the problem:

import asyncio

# This runs without an exception:
print(asyncio.Condition(asyncio.Lock()))

# This does not work:
async def example():
    print(asyncio.Condition(asyncio.Lock()))

# This raises "ValueError: loop argument must agree with lock":
asyncio.run(example())

And here's the relevant Janus code: https://github.com/aio-libs/janus/blob/d7970f8b76bcac2e087067ca4575ac845e481874/janus/init.py#L24-L42

```python class Queue(Generic[T]): def init(self, maxsize: int = 0) -> None: self._loop = current_loop() self._maxsize = maxsize

    self._init(maxsize)

    self._unfinished_tasks = 0

    self._sync_mutex = threading.Lock()
    self._sync_not_empty = threading.Condition(self._sync_mutex)
    self._sync_not_full = threading.Condition(self._sync_mutex)
    self._all_tasks_done = threading.Condition(self._sync_mutex)

    self._async_mutex = asyncio.Lock()
    # "loop argument must agree with lock" exception is raised here:
    self._async_not_empty = asyncio.Condition(self._async_mutex)
    self._async_not_full = asyncio.Condition(self._async_mutex)
    self._finished = asyncio.Event()
    self._finished.set()

```

Here's the code that raises that error: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/asyncio/locks.py#L219-L234

```python class Condition(_ContextManagerMixin, mixins._LoopBoundMixin): """Asynchronous equivalent to threading.Condition. This class implements condition variable objects. A condition variable allows one or more coroutines to wait until they are notified by another coroutine. A new Lock object is created and used as the underlying lock. """

def __init__(self, lock=None, *, loop=mixins._marker):
    super().__init__(loop=loop)
    if lock is None:
        lock = Lock()
    elif lock._loop is not self._get_loop():
        raise ValueError("loop argument must agree with lock")

```

Only mention I can find of that "loop argument must agree with lock" error is here - which doesn't have any tips for a workaround yet: https://giters.com/django/channels_redis/issues/278

So maybe this is an issue with Janus? I'm using https://pypi.org/project/janus/ 0.6.1 which is the latest release, from October 2020.

Running pytest -x --pdb helped me see this error:

File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 122, in dispatch_request await self.ds.refresh_schemas() File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 344, in refresh_schemas await self._refresh_schemas() File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 349, in _refresh_schemas await init_internal_db(internal_db) File "/Users/simon/Dropbox/Development/datasette/datasette/utils/internal_db.py", line 5, in init_internal_db await db.execute_write( File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 102, in execute_write return await self.execute_write_fn(_inner, block=block) File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 113, in execute_write_fn reply_queue = janus.Queue() File "/Users/simon/.local/share/virtualenvs/py310-Z8fTATkJ/lib/python3.10/site-packages/janus/__init__.py", line 39, in __init__ self._async_not_empty = asyncio.Condition(self._async_mutex) File "/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/asyncio/locks.py", line 234, in __init__ raise ValueError("loop argument must agree with lock") ValueError: loop argument must agree with lock

Then I created myself a temporary 3.10 environment using pipenv like so:

cd /tmp
mkdir py310
cd py310
pipenv shell --python /Users/simon/.pyenv/versions/3.10.0/bin/python

And used that with my Datasette checkout like so:

cd ~/.../datasette
pip install -e '.[test]'
pytest

The 3.10 tests failed a lot. Trying to run this locally:

``` /tmp % pyenv install 3.10 python-build: definition not found: 3.10

The following versions contain `3.10' in the name: 3.10.0a6 3.10-dev miniconda-3.10.1 miniconda3-3.10.1

See all available versions with `pyenv install --list'.

If the version you need is missing, try upgrading pyenv:

brew update && brew upgrade pyenv ``` So trying:

brew update && brew upgrade pyenv

Then did this:

/tmp % brew upgrade pyenv ==> Upgrading 1 outdated package: pyenv 1.2.24.1 -> 2.1.0 This decided to upgrade everything by downloaded everything on the internet. Aah, Homebrew.

But it looks like I have 3.10.0 available to pyenv now.

/tmp % pyenv install 3.10.0 python-build: use openssl@1.1 from homebrew python-build: use readline from homebrew Downloading Python-3.10.0.tar.xz... -> https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz Installing Python-3.10.0... ...