github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/issues/1843#issuecomment-1302679026 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1302679026 | IC_kwDOBm6k_c5NpU3y | 9599 | 2022-11-03T21:22:42Z | 2022-11-03T21:22:42Z | OWNER | Docs for the new `db.close()` method: https://docs.datasette.io/en/latest/internals.html#db-close | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1302678384 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1302678384 | IC_kwDOBm6k_c5NpUtw | 9599 | 2022-11-03T21:21:59Z | 2022-11-03T21:21:59Z | OWNER | I added extra debug info to `/-/threads` to see this for myself: ```diff diff --git a/datasette/app.py b/datasette/app.py index 02bd38f1..16579e28 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -969,6 +969,13 @@ class Datasette: "threads": [ {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads ], + "file_connections": { + db.name: [ + [dict(r) for r in conn.execute("pragma database_list").fetchall()] + for conn in db._all_file_connections + ] + for db in self.databases.values() + }, } # Only available in Python 3.7+ if hasattr(asyncio, "all_tasks"): ``` Output after hitting refresh on a few `/fixtures` tables to ensure more threads started: ``` "file_connections": { "_internal": [], "fixtures": [ [ { "seq": 0, "name": "main", "file": "/Users/simon/Dropbox/Development/datasette/fixtures.db" } ], [ { "seq": 0, "name": "main", "file": "/Users/simon/Dropbox/Development/datasette/fixtures.db" } ], [ { "seq": 0, "name": "main", "file": "/Users/simon/Dropbox/Development/datasette/fixtures.db" } ] ] }, ``` I decided not to ship this feature though as it leaks the names of internal database files. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1302634332 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1302634332 | IC_kwDOBm6k_c5NpJ9c | 9599 | 2022-11-03T20:34:56Z | 2022-11-03T20:34:56Z | OWNER | Confirmed that calling `conn.close()` on each SQLite file-based connection is the way to fix this problem. I'm adding a `db.close()` method (sync, not async - I tried async first but it was really hard to cause every thread in the pool to close its threadlocal database connection) which loops through all known open file-based connections and closes them. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1302574330 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1302574330 | IC_kwDOBm6k_c5No7T6 | 9599 | 2022-11-03T19:30:22Z | 2022-11-03T19:30:22Z | OWNER | This is affecting me a lot at the moment, on my laptop (runs fine in CI). Here's a change to `conftest.py` which highlights the problem - it cause a failure the moment there are more than 5 open files according to `psutil`: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index f4638a14..21d433c1 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,6 +1,7 @@ import httpx import os import pathlib +import psutil import pytest import re import subprocess @@ -192,3 +193,8 @@ def ds_unix_domain_socket_server(tmp_path_factory): yield ds_proc, uds # Shut it down at the end of the pytest session ds_proc.terminate() + + +def pytest_runtest_teardown(item: pytest.Item) -> None: + open_files = psutil.Process().open_files() + assert len(open_files) < 5 ``` The first error I get from this with `pytest --pdb -x` is here: ``` tests/test_api.py ............E >>>>> traceback >>>>> item = <Function test_sql_time_limit> def pytest_runtest_teardown(item: pytest.Item) -> None: open_files = psutil.Process().open_files() > assert len(open_files) < 5 E AssertionError: assert 5 < 5 E + where 5 = len([popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfglrt4p2/fixtures.db', fd=14), popenfile(... fd=19), popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmphdi5b250/fixtures.dot.db', fd=20)]) /Users/simon/Dropbox/Development/datasette/tests/conftest.py:200: AssertionError >>>>> entering PDB >>>>> >>>>> PDB post_mortem (IO-capturing turned off) >>>>> > /Users/simon/Dropbox/Development/datasette/tests/conftest.py(200)pytest_runtest_teardown() -> assert len(open_files) < 5 ``` That's this test: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/test_api.py#L656-L673 Which uses this fixture: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L228-L231 Which calls this func… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 |