{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1258129113", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1258129113, "node_id": "IC_kwDOBm6k_c5K_YbZ", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-09-26T14:30:11Z", "updated_at": "2022-09-26T14:48:31Z", "author_association": "CONTRIBUTOR", "body": "from your analysis, it seems like the GIL is blocking on loading of the data from sqlite to python, (particularly in the `fetchmany` call)\r\n\r\nthis is probably a simplistic idea, but what if you had the python code in the `execute` method iterate over the cursor and yield out rows or small chunks of rows.\r\n\r\nsomething like:             \r\n```python\r\n            with sqlite_timelimit(conn, time_limit_ms):\r\n                try:\r\n                    cursor = conn.cursor()\r\n                    cursor.execute(sql, params if params is not None else {})\r\n                except:\r\n                    ...\r\n            max_returned_rows = self.ds.max_returned_rows\r\n            if max_returned_rows == page_size:\r\n                max_returned_rows += 1\r\n                if max_returned_rows and truncate:\r\n                    for i, row in enumerate(cursor):\r\n                        yield row\r\n                        if i == max_returned_rows - 1:\r\n                            break\r\n                else:\r\n                    for row in cursor:\r\n                        yield row\r\n                    truncated = False                  \r\n```\r\n\r\nthis kind of thing works well with a postgres server side cursor, but i'm not sure if it will hold for sqlite. \r\n\r\nyou would still spend about the same amount of time in python and would be contending for the gil, but it would be could be non blocking.\r\n\r\ndepending on the data flow, this could also some benefit for memory. (data stays in more compact sqlite-land until you need it)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}