{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-604667029", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 604667029, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNDY2NzAyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-26T20:26:46Z", "updated_at": "2020-03-26T20:26:46Z", "author_association": "OWNER", "body": "I think I can tell what the current file limit is like so:\r\n```\r\nIn [1]: import resource                                                                                                                                                                   \r\n\r\nIn [2]: resource.getrlimit(resource.RLIMIT_NOFILE)                                                                                                                                        \r\nOut[2]: (256, 9223372036854775807)\r\n```\r\nSo maybe I should have Datasette refuse to open more database files than that number minus 5 (to give me some spare room for opening CSS files etc).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-604665229", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 604665229, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNDY2NTIyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-26T20:22:48Z", "updated_at": "2020-03-26T20:22:48Z", "author_association": "OWNER", "body": "I also eventually get this error:\r\n```\r\nTraceback (most recent call last):\r\n  File \"/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py\", line 121, in route_path\r\n    return await view(new_scope, receive, send)\r\n  File \"/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py\", line 336, in inner_static\r\n    await asgi_send_file(send, full_path, chunk_size=chunk_size)\r\n  File \"/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py\", line 303, in asgi_send_file\r\n    async with aiofiles.open(str(filepath), mode=\"rb\") as fp:\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/aiofiles/base.py\", line 78, in __aenter__\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/aiofiles/threadpool/__init__.py\", line 35, in _open\r\n  File \"/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py\", line 57, in run\r\nOSError: [Errno 24] Too many open files: '/Users/simonw/Dropbox/Development/datasette/datasette/static/app.css'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-604569063", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 604569063, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNDU2OTA2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-26T17:32:06Z", "updated_at": "2020-03-26T17:32:06Z", "author_association": "OWNER", "body": "While running it against a nested directory with a TON of databases I kept seeing errors like this:\r\n```\r\nTraceback (most recent call last):\r\n  File \"/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py\", line 121, in route_path\r\n    return await view(new_scope, receive, send)\r\n  File \"/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py\", line 193, in view\r\n    request, **scope[\"url_route\"][\"kwargs\"]\r\n  File \"/Users/simonw/Dropbox/Development/datasette/datasette/views/index.py\", line 58, in get\r\n    tables[table][\"num_relationships_for_sorting\"] = count\r\nKeyError: 'primary-candidates-2018/rep_candidates'\r\n```\r\nAnd\r\n```\r\nTraceback (most recent call last):\r\n  File \"/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py\", line 121, in route_path\r\n    return await view(new_scope, receive, send)\r\n  File \"/Users/simonw/Dropbox/Development/datasette/datasette/utils/asgi.py\", line 193, in view\r\n    request, **scope[\"url_route\"][\"kwargs\"]\r\n  File \"/Users/simonw/Dropbox/Development/datasette/datasette/views/index.py\", line 58, in get\r\n    tables[table][\"num_relationships_for_sorting\"] = count\r\nKeyError: 'space_used'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-604561639", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 604561639, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNDU2MTYzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-26T17:22:07Z", "updated_at": "2020-03-26T17:22:07Z", "author_association": "OWNER", "body": "Here's the new utility function I should be using to verify database files that I find: \r\n\r\nhttps://github.com/simonw/datasette/blob/6aa516d82dea9885cb4db8d56ec2ccfd4cd9b840/datasette/utils/__init__.py#L773-L787", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586441484", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586441484, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjQ0MTQ4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T19:34:25Z", "updated_at": "2020-02-14T19:34:25Z", "author_association": "OWNER", "body": "I've figured out how to tell if a database is safe to open or not:\r\n```sql\r\nselect sql from sqlite_master where sql like 'CREATE VIRTUAL TABLE%';\r\n```\r\nThis returns the SQL definitions for virtual tables. The bit after `using` tells you what they need.\r\n\r\nRun this against a SpatiaLite database and you get the following:\r\n```sql\r\nCREATE VIRTUAL TABLE SpatialIndex USING VirtualSpatialIndex()\r\nCREATE VIRTUAL TABLE ElementaryGeometries USING VirtualElementary()\r\n```\r\nRun it against an Apple Photos `photos.db` file (found with `find ~/Library | grep photos.db`) and you get this (partial list):\r\n```sql\r\nCREATE VIRTUAL TABLE RidList_VirtualReader using RidList_VirtualReaderModule\r\nCREATE VIRTUAL TABLE Array_VirtualReader using Array_VirtualReaderModule\r\nCREATE VIRTUAL TABLE LiGlobals_VirtualBufferReader using VirtualBufferReaderModule\r\nCREATE VIRTUAL TABLE RKPlace_RTree using rtree (modelId,minLongitude,maxLongitude,minLatitude,maxLatitude)\r\n```\r\nFor a database with FTS4 you get:\r\n```sql\r\nCREATE VIRTUAL TABLE \"docs_fts\" USING FTS4 (\r\n    [title], [content], content=\"docs\"\r\n)\r\n```\r\nFTS5:\r\n```sql\r\nCREATE VIRTUAL TABLE [FARA_All_Registrants_fts] USING FTS5 (\r\n                [Name], [Address_1], [Address_2],\r\n                content=[FARA_All_Registrants]\r\n            )\r\n```\r\nSo I can use this to figure out all of the `using` pieces and then compare them to a list of known support ones.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586112662", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586112662, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjExMjY2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T06:05:27Z", "updated_at": "2020-02-14T06:05:27Z", "author_association": "OWNER", "body": "It think the fix is to use an old-fashioned `threading` module daemon thread directly. That should exit cleanly when the program exits.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586111619", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586111619, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjExMTYxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T06:01:24Z", "updated_at": "2020-02-14T06:01:24Z", "author_association": "OWNER", "body": "https://gist.github.com/clchiou/f2608cbe54403edb0b13 might work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586111102", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586111102, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjExMTEwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T05:59:24Z", "updated_at": "2020-02-14T06:00:36Z", "author_association": "OWNER", "body": "Interesting new problem: hitting Ctrl+C no longer terminates the problem provided that `scan_dirs()` thread is still running.\r\n\r\nhttps://stackoverflow.com/questions/49992329/the-workers-in-threadpoolexecutor-is-not-really-daemon has clues. The workers are only meant to exit when their worker queues are empty.\r\n\r\nBut... I want to run the worker every 10 seconds. How do I do that without having it loop forever and hence never quit?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586109784", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586109784, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjEwOTc4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T05:53:50Z", "updated_at": "2020-02-14T05:54:21Z", "author_association": "OWNER", "body": "... cheating like this seems to work:\r\n```\r\nfor name, db in list(self.ds.databases.items()):\r\n```\r\nPython built-in operations are supposedly threadsafe, so in this case I can grab a copy of the list atomically (I think) and then safely iterate over it.\r\n\r\nSeems to work in my testing. Wish I could prove it with a unit test though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586109238", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586109238, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjEwOTIzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T05:51:12Z", "updated_at": "2020-02-14T05:51:12Z", "author_association": "OWNER", "body": "... or maybe I can cheat and wrap the access to `self.ds.databases.items()` in `list()`, so I'm iterating over an atomically-created list of those things instead? I'll try that first.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586109032", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586109032, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjEwOTAzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T05:50:15Z", "updated_at": "2020-02-14T05:50:15Z", "author_association": "OWNER", "body": "So I need to ensure the `ds.databases` data structure is manipulated in a thread-safe manner.\r\n\r\nMainly I need to ensure that it is locked during iterations over it, then unlocked at the end.\r\n\r\nTrickiest part is probably ensuring there is a test that proves this is working - I feel like I got lucky encountering that `RuntimeError` as early as I did.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586107989", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586107989, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjEwNzk4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T05:45:12Z", "updated_at": "2020-02-14T05:45:12Z", "author_association": "OWNER", "body": "I tried running the `scan_dirs()` method in a thread and got an interesting error while trying to load the homepage: `RuntimeError: OrderedDict mutated during iteration`\r\n\r\nMakes sense - I had a thread that added an item to that dictionary right while the homepage was attempting to run this code:\r\n\r\nhttps://github.com/simonw/datasette/blob/efa54b439fd0394440c302602b919255047b59c5/datasette/views/index.py#L24-L27\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586069529", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586069529, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjA2OTUyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T02:37:17Z", "updated_at": "2020-02-14T02:37:17Z", "author_association": "OWNER", "body": "Another problem: if any of the found databases use SpatiaLite then Datasette will fail to start at all.\r\n\r\nIt should skip them instead.\r\n\r\nThe `select * from sqlite_master` check apparently isn't quite enough to catch this case.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586068095", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586068095, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjA2ODA5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T02:30:37Z", "updated_at": "2020-02-14T02:30:46Z", "author_association": "OWNER", "body": "This can take a LONG time to run, and at the moment it's blocking and prevents Datasette from starting up.\r\n\r\nIt would be much better if this ran in a thread, or an asyncio task. Probably have to be a thread because there's no easy `async` version of `pathlib.Path.glob()` that I've seen.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/672#issuecomment-586067794", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/672", "id": 586067794, "node_id": "MDEyOklzc3VlQ29tbWVudDU4NjA2Nzc5NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-14T02:29:16Z", "updated_at": "2020-02-14T02:29:16Z", "author_association": "OWNER", "body": "One design issue: how to pick neat unique names for database files in a file hierarchy?\r\n\r\nHere's what I have so far:\r\n\r\nhttps://github.com/simonw/datasette/blob/fe6f9e6a7397cab2e4bc57745a8da9d824dad218/datasette/app.py#L231-L237\r\n\r\nFor these files:\r\n```\r\n../travel-old.db\r\n../sf-tree-history/trees.db\r\n../library-of-congress/records-from-df.db\r\n```\r\nIt made these names:\r\n```\r\ntravel-old\r\nsf-tree-history_trees\r\nlibrary-of-congress_records-from-df\r\n```\r\nMaybe this is good enough? Needs some tests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 565064079, "label": "--dirs option for scanning directories for SQLite databases"}, "performed_via_github_app": null}