{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-781560865", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 781560865, "node_id": "MDEyOklzc3VlQ29tbWVudDc4MTU2MDg2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-18T18:50:38Z", "updated_at": "2021-02-18T18:50:38Z", "author_association": "OWNER", "body": "I started trying to use locks to resolve this but I've not figured out the right way to do that yet - here's my first experiment:\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 9e15a16..1681c9d 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -217,6 +217,7 @@ class Datasette:\r\n         self.inspect_data = inspect_data\r\n         self.immutables = set(immutables or [])\r\n         self.databases = collections.OrderedDict()\r\n+        self._refresh_schemas_lock = threading.Lock()\r\n         if memory or not self.files:\r\n             self.add_database(Database(self, is_memory=True), name=\"_memory\")\r\n         # memory_name is a random string so that each Datasette instance gets its own\r\n@@ -324,6 +325,13 @@ class Datasette:\r\n         self.client = DatasetteClient(self)\r\n \r\n     async def refresh_schemas(self):\r\n+        return\r\n+        if self._refresh_schemas_lock.locked():\r\n+            return\r\n+        with self._refresh_schemas_lock:\r\n+            await self._refresh_schemas()\r\n+\r\n+    async def _refresh_schemas(self):\r\n         internal_db = self.databases[\"_internal\"]\r\n         if not self.internal_db_created:\r\n             await init_internal_db(internal_db)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-781560989", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 781560989, "node_id": "MDEyOklzc3VlQ29tbWVudDc4MTU2MDk4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-18T18:50:53Z", "updated_at": "2021-02-18T18:50:53Z", "author_association": "OWNER", "body": "Ideally I'd figure out a way to replicate this error in a concurrent unit test.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881204343", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881204343, "node_id": "IC_kwDOBm6k_c40hhx3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T06:13:11Z", "updated_at": "2021-07-16T06:13:11Z", "author_association": "OWNER", "body": "This just broke the `datasette-graphql` test suite: https://github.com/simonw/datasette-graphql/issues/77 - I need to figure out a solution here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881204782", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881204782, "node_id": "IC_kwDOBm6k_c40hh4u", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T06:14:12Z", "updated_at": "2021-07-16T06:14:12Z", "author_association": "OWNER", "body": "Here's the traceback I got from `datasette-graphql` (annoyingly only running the tests in GitHub Actions CI - I've not been able to replicate on my laptop yet):\r\n\r\n```\r\ntests/test_utils.py .                                                    [100%]\r\n\r\n=================================== FAILURES ===================================\r\n_________________________ test_graphql_examples[path0] _________________________\r\n\r\nds = <datasette.app.Datasette object at 0x7f6b8b6f8fd0>\r\npath = PosixPath('/home/runner/work/datasette-graphql/datasette-graphql/examples/filters.md')\r\n\r\n    @pytest.mark.asyncio\r\n    @pytest.mark.parametrize(\r\n        \"path\", (pathlib.Path(__file__).parent.parent / \"examples\").glob(\"*.md\")\r\n    )\r\n    async def test_graphql_examples(ds, path):\r\n        content = path.read_text()\r\n        query = graphql_re.search(content)[1]\r\n        try:\r\n            variables = variables_re.search(content)[1]\r\n        except TypeError:\r\n            variables = \"{}\"\r\n        expected = json.loads(json_re.search(content)[1])\r\n        response = await ds.client.post(\r\n            \"/graphql\",\r\n            json={\r\n                \"query\": query,\r\n                \"variables\": json.loads(variables),\r\n            },\r\n        )\r\n>       assert response.status_code == 200, response.json()\r\nE       AssertionError: {'data': {'repos_arraycontains': None, 'users_contains': None, 'users_date': None, 'users_endswith': None, ...}, 'erro...\", 'path': ['users_gt']}, {'locations': [{'column': 5, 'line': 34}], 'message': \"'rows'\", 'path': ['users_gte']}, ...]}\r\nE       assert 500 == 200\r\nE        +  where 500 = <Response [500 Internal Server Error]>.status_code\r\n\r\ntests/test_graphql.py:142: AssertionError\r\n----------------------------- Captured stderr call -----------------------------\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\nTraceback (most recent call last):\r\n  File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py\", line 1171, in route_path\r\n    response = await view(request, send)\r\n  File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py\", line 151, in view\r\n    request, **request.scope[\"url_route\"][\"kwargs\"]\r\n  File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py\", line 123, in dispatch_request\r\n    await self.ds.refresh_schemas()\r\n  File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py\", line 338, in refresh_schemas\r\n    await init_internal_db(internal_db)\r\n  File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/utils/internal_db.py\", line 16, in init_internal_db\r\n    block=True,\r\n  File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py\", line 102, in execute_write\r\n    return await self.execute_write_fn(_inner, block=block)\r\n  File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py\", line 118, in execute_write_fn\r\n    raise result\r\n  File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py\", line 139, in _execute_writes\r\n    result = task.fn(conn)\r\n  File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py\", line 100, in _inner\r\n    return conn.execute(sql, params or [])\r\nsqlite3.OperationalError: table databases already exists\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881663968", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881663968, "node_id": "IC_kwDOBm6k_c40jR_g", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:18:42Z", "updated_at": "2021-07-16T19:18:42Z", "author_association": "OWNER", "body": "The race condition happens inside this method - initially with the call to `await init_internal_db()`: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/app.py#L334-L359", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881664408", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881664408, "node_id": "IC_kwDOBm6k_c40jSGY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:19:35Z", "updated_at": "2021-07-16T19:19:35Z", "author_association": "OWNER", "body": "The only place that calls `refresh_schemas()` is here: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/views/base.py#L120-L124\r\n\r\nIdeally only one call to `refresh_schemas()` would be running at any one time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881665383", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881665383, "node_id": "IC_kwDOBm6k_c40jSVn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:21:35Z", "updated_at": "2021-07-16T19:21:35Z", "author_association": "OWNER", "body": "https://stackoverflow.com/a/25799871/6083 has a good example of using `asyncio.Lock()`:\r\n\r\n```python\r\nstuff_lock = asyncio.Lock()\r\n\r\nasync def get_stuff(url):\r\n    async with stuff_lock:\r\n        if url in cache:\r\n            return cache[url]\r\n        stuff = await aiohttp.request('GET', url)\r\n        cache[url] = stuff\r\n        return stuff\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881668759", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881668759, "node_id": "IC_kwDOBm6k_c40jTKX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:27:46Z", "updated_at": "2021-07-16T19:27:46Z", "author_association": "OWNER", "body": "Second attempt at this:\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 5976d8b..5f348cb 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -224,6 +224,7 @@ class Datasette:\r\n         self.inspect_data = inspect_data\r\n         self.immutables = set(immutables or [])\r\n         self.databases = collections.OrderedDict()\r\n+        self._refresh_schemas_lock = asyncio.Lock()\r\n         self.crossdb = crossdb\r\n         if memory or crossdb or not self.files:\r\n             self.add_database(Database(self, is_memory=True), name=\"_memory\")\r\n@@ -332,6 +333,12 @@ class Datasette:\r\n         self.client = DatasetteClient(self)\r\n \r\n     async def refresh_schemas(self):\r\n+        if self._refresh_schemas_lock.locked():\r\n+            return\r\n+        async with self._refresh_schemas_lock:\r\n+            await self._refresh_schemas()\r\n+\r\n+    async def _refresh_schemas(self):\r\n         internal_db = self.databases[\"_internal\"]\r\n         if not self.internal_db_created:\r\n             await init_internal_db(internal_db)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881671706", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881671706, "node_id": "IC_kwDOBm6k_c40jT4a", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:32:05Z", "updated_at": "2021-07-16T19:32:05Z", "author_association": "OWNER", "body": "The test suite passes with that change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881674857", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881674857, "node_id": "IC_kwDOBm6k_c40jUpp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:38:39Z", "updated_at": "2021-07-16T19:38:39Z", "author_association": "OWNER", "body": "I can't replicate the race condition locally with or without this patch. I'm going to push the commit and then test the CI run from `datasette-graphql` that was failing against it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881677620", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881677620, "node_id": "IC_kwDOBm6k_c40jVU0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:44:12Z", "updated_at": "2021-07-16T19:44:12Z", "author_association": "OWNER", "body": "That fixed the race condition in the `datasette-graphql` tests, which is the only place that I've been able to successfully replicate this. I'm going to land this change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null}