html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/issues/1843#issuecomment-1278478042,https://api.github.com/repos/simonw/datasette/issues/1843,1278478042,IC_kwDOBm6k_c5MNAba,9599,2022-10-14T04:46:29Z,2022-10-14T04:46:29Z,OWNER,"I did `pip install psutil` and then ran this in the debugger for one of these errors: ```python import psutil for f in psutil.Process().open_files(): print(f) ``` The output looked like this: ``` popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=11) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=12) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfd3oyo10/fixtures.dot.db', fd=13) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpezwfu7w8/fixtures.db', fd=14) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=15) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=16) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/fixtures.db', fd=17) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=18) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdp4we7hb/fixtures.db', fd=19) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfd3oyo10/fixtures.dot.db', fd=20) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4ljq_ai0/fixtures.db', fd=21) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpezwfu7w8/fixtures.db', fd=22) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/fixtures.db', fd=24) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/extra database.db', fd=25) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/fixtures.db', fd=26) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/fixtures.db', fd=27) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdp4we7hb/fixtures.db', fd=28) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp42e6vyj_/fixtures.db', fd=29) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4ljq_ai0/fixtures.db', fd=31) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpw32vwkjq/fixtures.db', fd=32) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/extra database.db', fd=33) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/fixtures.db', fd=34) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpw32vwkjq/fixtures.db', fd=35) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/foo-bar.db', fd=36) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/foo.db', fd=37) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/fixtures.db', fd=38) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpcl1edmyv/fixtures.db', fd=39) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp42e6vyj_/fixtures.db', fd=40) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=41) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0w5jugqk/fixtures.db', fd=42) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmb9y9fba/fixtures.db', fd=43) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/fixtures.db', fd=44) popenfile(path='/Users/simon/Dropbox/Development/datasette/tests/spatialite.db', fd=45) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0w5jugqk/fixtures.db', fd=46) popenfile(path='/Users/simon/Dropbox/Development/datasette/tests/spatialite.db', fd=47) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmb9y9fba/fixtures.db', fd=48) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_create0/does_not_exist_yet.db', fd=49) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/data.db', fd=92) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/fixtures.db', fd=93) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/data.db', fd=94) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/db.db', fd=95) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/nested/db.db', fd=99) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_deduplicate_same_da0/db.db', fd=100) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=101) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=102) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/db.db', fd=103) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/nested/db.db', fd=104) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=105) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=106) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=107) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=109) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=111) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=113) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=117) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpky1jamnv/fixtures.db', fd=118) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=119) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/demo.db', fd=120) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/immutable.db', fd=121) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/k.sqlite', fd=122) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/j.sqlite3', fd=123) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=124) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/extra database.db', fd=125) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=126) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/extra database.db', fd=127) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=128) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=129) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=130) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=131) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=132) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=133) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=134) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=135) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=136) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=137) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=138) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=139) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=140) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=141) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=142) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=143) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=144) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=145) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=146) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_10.db', fd=147) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmzu4jbdx/fixtures.db', fd=148) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpskyh32wh/fixtures.db', fd=149) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp966705ec/fixtures.db', fd=150) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=151) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=152) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=153) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=154) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=155) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=156) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=157) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=158) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=159) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=160) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_10.db', fd=161) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=162) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=163) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=164) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=165) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=166) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=167) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=168) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=169) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=170) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=171) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpky1jamnv/fixtures.db', fd=172) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwmewpy8e/fixtures.db', fd=173) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmzu4jbdx/fixtures.db', fd=174) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpkyn60obe/fixtures.db', fd=175) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpskyh32wh/fixtures.db', fd=176) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpcl1edmyv/fixtures.db', fd=177) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp966705ec/fixtures.db', fd=178) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=179) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwmewpy8e/fixtures.db', fd=180) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=181) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpkyn60obe/fixtures.db', fd=182) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=183) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/fixtures.db', fd=184) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf3_6i1d6/fixtures.db', fd=185) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp6_9vzq4o/fixtures.db', fd=187) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpqtubcoah/fixtures.db', fd=188) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/extra database.db', fd=189) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/fixtures.db', fd=190) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2gauogmy/fixtures.db', fd=191) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf3_6i1d6/fixtures.db', fd=192) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/fixtures.db', fd=193) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp6_9vzq4o/fixtures.db', fd=197) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/extra database.db', fd=198) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpqtubcoah/fixtures.db', fd=211) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/fixtures.db', fd=212) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2gauogmy/fixtures.db', fd=215) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpd2tleqni/fixtures.db', fd=216) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpd2tleqni/fixtures.db', fd=217) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmplpdc3wfl/fixtures.db', fd=219) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdnsscl5v/fixtures.db', fd=222) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmptjayd3pn/fixtures.db', fd=223) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4u5oo5ne/fixtures.db', fd=224) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0dgqc4wx/fixtures.db', fd=225) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdnsscl5v/fixtures.db', fd=226) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoehh1_tg/fixtures.db', fd=227) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4u5oo5ne/fixtures.db', fd=247) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0dgqc4wx/fixtures.db', fd=251) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoehh1_tg/fixtures.db', fd=253) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmptjayd3pn/fixtures.db', fd=254) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test-view-names0/fixtures.db', fd=255) ``` Clearly something is bad with the way fixtures work in the tests - a huge number of `fixtures.db` database files are being created and left open!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1278480437,https://api.github.com/repos/simonw/datasette/issues/1843,1278480437,IC_kwDOBm6k_c5MNBA1,9599,2022-10-14T04:51:10Z,2022-10-14T04:51:10Z,OWNER,Extracted a TIL: https://til.simonwillison.net/python/too-many-open-files-psutil,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1278537920,https://api.github.com/repos/simonw/datasette/issues/1843,1278537920,IC_kwDOBm6k_c5MNPDA,9599,2022-10-14T06:19:55Z,2022-10-14T06:20:06Z,OWNER,Maybe I need to explicitly close those SQLite connections held by the Datasette instance after this line: https://github.com/simonw/datasette/blob/79aa0de083d38a9975915d5a4cc68ca6c74fbe3d/tests/fixtures.py#L165,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1288214953,https://api.github.com/repos/simonw/datasette/issues/1843,1288214953,IC_kwDOBm6k_c5MyJmp,9599,2022-10-23T22:22:52Z,2022-10-23T22:22:52Z,OWNER,This seems to have fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1291466613,https://api.github.com/repos/simonw/datasette/issues/1843,1291466613,IC_kwDOBm6k_c5M-jd1,9599,2022-10-26T04:02:56Z,2022-10-26T04:02:56Z,OWNER,Just saw this error again!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1291467084,https://api.github.com/repos/simonw/datasette/issues/1843,1291467084,IC_kwDOBm6k_c5M-jlM,9599,2022-10-26T04:03:49Z,2022-10-26T04:03:49Z,OWNER,"This time I'm suspicious that there are open SQLite files tucked away in thread locals hidden inside my thread pool executor: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L24 https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L204-L214","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1302574330,https://api.github.com/repos/simonw/datasette/issues/1843,1302574330,IC_kwDOBm6k_c5No7T6,9599,2022-11-03T19:30:22Z,2022-11-03T19:30:22Z,OWNER,"This is affecting me a lot at the moment, on my laptop (runs fine in CI). Here's a change to `conftest.py` which highlights the problem - it cause a failure the moment there are more than 5 open files according to `psutil`: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index f4638a14..21d433c1 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,6 +1,7 @@ import httpx import os import pathlib +import psutil import pytest import re import subprocess @@ -192,3 +193,8 @@ def ds_unix_domain_socket_server(tmp_path_factory): yield ds_proc, uds # Shut it down at the end of the pytest session ds_proc.terminate() + + +def pytest_runtest_teardown(item: pytest.Item) -> None: + open_files = psutil.Process().open_files() + assert len(open_files) < 5 ``` The first error I get from this with `pytest --pdb -x` is here: ``` tests/test_api.py ............E >>>>> traceback >>>>> item = def pytest_runtest_teardown(item: pytest.Item) -> None: open_files = psutil.Process().open_files() > assert len(open_files) < 5 E AssertionError: assert 5 < 5 E + where 5 = len([popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfglrt4p2/fixtures.db', fd=14), popenfile(... fd=19), popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmphdi5b250/fixtures.dot.db', fd=20)]) /Users/simon/Dropbox/Development/datasette/tests/conftest.py:200: AssertionError >>>>> entering PDB >>>>> >>>>> PDB post_mortem (IO-capturing turned off) >>>>> > /Users/simon/Dropbox/Development/datasette/tests/conftest.py(200)pytest_runtest_teardown() -> assert len(open_files) < 5 ``` That's this test: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/test_api.py#L656-L673 Which uses this fixture: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L228-L231 Which calls this function: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L105-L122 So now I'm suspicious that, even though the fixture is meant to be session scoped, the way I'm using `with tempfile.TemporaryDirectory() as tmpdir:` is causing a whole load of files to be created and held open which are not later closed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1302634332,https://api.github.com/repos/simonw/datasette/issues/1843,1302634332,IC_kwDOBm6k_c5NpJ9c,9599,2022-11-03T20:34:56Z,2022-11-03T20:34:56Z,OWNER,"Confirmed that calling `conn.close()` on each SQLite file-based connection is the way to fix this problem. I'm adding a `db.close()` method (sync, not async - I tried async first but it was really hard to cause every thread in the pool to close its threadlocal database connection) which loops through all known open file-based connections and closes them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1302678384,https://api.github.com/repos/simonw/datasette/issues/1843,1302678384,IC_kwDOBm6k_c5NpUtw,9599,2022-11-03T21:21:59Z,2022-11-03T21:21:59Z,OWNER,"I added extra debug info to `/-/threads` to see this for myself: ```diff diff --git a/datasette/app.py b/datasette/app.py index 02bd38f1..16579e28 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -969,6 +969,13 @@ class Datasette: ""threads"": [ {""name"": t.name, ""ident"": t.ident, ""daemon"": t.daemon} for t in threads ], + ""file_connections"": { + db.name: [ + [dict(r) for r in conn.execute(""pragma database_list"").fetchall()] + for conn in db._all_file_connections + ] + for db in self.databases.values() + }, } # Only available in Python 3.7+ if hasattr(asyncio, ""all_tasks""): ``` Output after hitting refresh on a few `/fixtures` tables to ensure more threads started: ``` ""file_connections"": { ""_internal"": [], ""fixtures"": [ [ { ""seq"": 0, ""name"": ""main"", ""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db"" } ], [ { ""seq"": 0, ""name"": ""main"", ""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db"" } ], [ { ""seq"": 0, ""name"": ""main"", ""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db"" } ] ] }, ``` I decided not to ship this feature though as it leaks the names of internal database files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1302679026,https://api.github.com/repos/simonw/datasette/issues/1843,1302679026,IC_kwDOBm6k_c5NpU3y,9599,2022-11-03T21:22:42Z,2022-11-03T21:22:42Z,OWNER,Docs for the new `db.close()` method: https://docs.datasette.io/en/latest/internals.html#db-close,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1353520615,https://api.github.com/repos/simonw/datasette/issues/1843,1353520615,IC_kwDOBm6k_c5QrRXn,9599,2022-12-15T18:19:25Z,2022-12-15T18:19:25Z,OWNER,"I've been seeing this error again: ``` ERROR tests/test_api_write.py::test_create_table[input16-400-expected_response16] - OSError: [Errno 24] Too ... ERROR tests/test_api_write.py::test_create_table[input17-400-expected_response17] - OSError: [Errno 24] Too ... ERROR tests/test_api_write.py::test_create_table[input18-400-expected_response18] - OSError: [Errno 24] Too ... ``` It doesn't happen in CI, and it turns out that's because CI runs `pytest -n auto` which splits the tests across multiple parallel processes. I've been working around the error on my laptop using `pytest -n auto` there too, but I'd rather not have to do that. This is also getting in my way when I try to debug other issues, like this one: - #1955","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1353522211,https://api.github.com/repos/simonw/datasette/issues/1843,1353522211,IC_kwDOBm6k_c5QrRwj,9599,2022-12-15T18:21:02Z,2022-12-15T18:21:02Z,OWNER,"When I initially built this test suite Datasette didn't have the `memory_name=` mechanism for creating persistent in-memory databases. I'm going to see if I can switch to that for the majority of Datasette's tests. Might find that doing so both fixes this ""too many open files"" issue AND gives me a significant speed improvement to the test site too! Hopefully I can do most of the work on that in this big ugly function: https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/fixtures.py#L104-L173","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1353522652,https://api.github.com/repos/simonw/datasette/issues/1843,1353522652,IC_kwDOBm6k_c5QrR3c,9599,2022-12-15T18:21:27Z,2022-12-15T18:21:27Z,OWNER,"I'll still use on-disk test databases for `is_immutable=True`, but not for the majority of tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1353690591,https://api.github.com/repos/simonw/datasette/issues/1843,1353690591,IC_kwDOBm6k_c5Qr63f,9599,2022-12-15T20:49:05Z,2022-12-15T20:49:05Z,OWNER,"I have a nasty feeling the cleaner solution for this would involve porting my entire test suite from `def test_blah(app_client)` sync functions (which work due to a `@async_to_sync` call in `TestClient`) to `async def test_blah(async_fixture):` functions instead. I've been using that latter pattern for new tests (and plugin tests) for quite a while now, but I never took on the job of refactoring all of the old ones. A search for `(app_client):` across the whole project currently returns 194 results which might be a reasonable target to try switching to the new pattern as a starting point. No idea if it will have much impact on the ""Too many open files"" errors though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1356480256,https://api.github.com/repos/simonw/datasette/issues/1843,1356480256,IC_kwDOBm6k_c5Q2j8A,9599,2022-12-17T22:01:13Z,2022-12-17T22:01:13Z,OWNER,"The refactor in the following issue did NOT prevent this error from occurring when I try to run the full `pytest` suite on my Mac laptop: - #1959","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1356481595,https://api.github.com/repos/simonw/datasette/issues/1843,1356481595,IC_kwDOBm6k_c5Q2kQ7,9599,2022-12-17T22:02:40Z,2022-12-17T22:02:40Z,OWNER,"This is interesting: ```python import resource print(resource.getrlimit(resource.RLIMIT_NOFILE)) ``` On Mac: ```pycon >>> print(resource.getrlimit(resource.RLIMIT_NOFILE)) (256, 9223372036854775807) ``` On Ubuntu (in a Docker `ubuntu:22.04` container): ```pycon >>> resource.getrlimit(resource.RLIMIT_NOFILE) (1048576, 1048576) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,