html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/1843#issuecomment-1302679026,https://api.github.com/repos/simonw/datasette/issues/1843,1302679026,IC_kwDOBm6k_c5NpU3y,9599,2022-11-03T21:22:42Z,2022-11-03T21:22:42Z,OWNER,Docs for the new `db.close()` method: https://docs.datasette.io/en/latest/internals.html#db-close,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,
https://github.com/simonw/datasette/issues/1843#issuecomment-1302678384,https://api.github.com/repos/simonw/datasette/issues/1843,1302678384,IC_kwDOBm6k_c5NpUtw,9599,2022-11-03T21:21:59Z,2022-11-03T21:21:59Z,OWNER,"I added extra debug info to `/-/threads` to see this for myself:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 02bd38f1..16579e28 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -969,6 +969,13 @@ class Datasette:
""threads"": [
{""name"": t.name, ""ident"": t.ident, ""daemon"": t.daemon} for t in threads
],
+ ""file_connections"": {
+ db.name: [
+ [dict(r) for r in conn.execute(""pragma database_list"").fetchall()]
+ for conn in db._all_file_connections
+ ]
+ for db in self.databases.values()
+ },
}
# Only available in Python 3.7+
if hasattr(asyncio, ""all_tasks""):
```
Output after hitting refresh on a few `/fixtures` tables to ensure more threads started:
```
""file_connections"": {
""_internal"": [],
""fixtures"": [
[
{
""seq"": 0,
""name"": ""main"",
""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db""
}
],
[
{
""seq"": 0,
""name"": ""main"",
""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db""
}
],
[
{
""seq"": 0,
""name"": ""main"",
""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db""
}
]
]
},
```
I decided not to ship this feature though as it leaks the names of internal database files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,
https://github.com/simonw/datasette/issues/1843#issuecomment-1302634332,https://api.github.com/repos/simonw/datasette/issues/1843,1302634332,IC_kwDOBm6k_c5NpJ9c,9599,2022-11-03T20:34:56Z,2022-11-03T20:34:56Z,OWNER,"Confirmed that calling `conn.close()` on each SQLite file-based connection is the way to fix this problem.
I'm adding a `db.close()` method (sync, not async - I tried async first but it was really hard to cause every thread in the pool to close its threadlocal database connection) which loops through all known open file-based connections and closes them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,
https://github.com/simonw/datasette/issues/1843#issuecomment-1302574330,https://api.github.com/repos/simonw/datasette/issues/1843,1302574330,IC_kwDOBm6k_c5No7T6,9599,2022-11-03T19:30:22Z,2022-11-03T19:30:22Z,OWNER,"This is affecting me a lot at the moment, on my laptop (runs fine in CI).
Here's a change to `conftest.py` which highlights the problem - it cause a failure the moment there are more than 5 open files according to `psutil`:
```diff
diff --git a/tests/conftest.py b/tests/conftest.py
index f4638a14..21d433c1 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,6 +1,7 @@
import httpx
import os
import pathlib
+import psutil
import pytest
import re
import subprocess
@@ -192,3 +193,8 @@ def ds_unix_domain_socket_server(tmp_path_factory):
yield ds_proc, uds
# Shut it down at the end of the pytest session
ds_proc.terminate()
+
+
+def pytest_runtest_teardown(item: pytest.Item) -> None:
+ open_files = psutil.Process().open_files()
+ assert len(open_files) < 5
```
The first error I get from this with `pytest --pdb -x` is here:
```
tests/test_api.py ............E
>>>>> traceback >>>>>
item =
def pytest_runtest_teardown(item: pytest.Item) -> None:
open_files = psutil.Process().open_files()
> assert len(open_files) < 5
E AssertionError: assert 5 < 5
E + where 5 = len([popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfglrt4p2/fixtures.db', fd=14), popenfile(... fd=19), popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmphdi5b250/fixtures.dot.db', fd=20)])
/Users/simon/Dropbox/Development/datasette/tests/conftest.py:200: AssertionError
>>>>> entering PDB >>>>>
>>>>> PDB post_mortem (IO-capturing turned off) >>>>>
> /Users/simon/Dropbox/Development/datasette/tests/conftest.py(200)pytest_runtest_teardown()
-> assert len(open_files) < 5
```
That's this test:
https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/test_api.py#L656-L673
Which uses this fixture:
https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L228-L231
Which calls this function:
https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L105-L122
So now I'm suspicious that, even though the fixture is meant to be session scoped, the way I'm using `with tempfile.TemporaryDirectory() as tmpdir:` is causing a whole load of files to be created and held open which are not later closed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,