issue_comments: 1724298817

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions issue performed_via_github_app
https://github.com/simonw/datasette/issues/2189#issuecomment-1724298817 https://api.github.com/repos/simonw/datasette/issues/2189 1724298817 IC_kwDOBm6k_c5mxrZB 9599 2023-09-18T20:11:26Z 2023-09-18T20:11:26Z OWNER

Now that I've confirmed that parallel query execution of the kind introduced in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 can cause hangs (presumably some kind of locking issue) against in-memory databases, some options:

  1. Disable parallel execution entirely and rip out related code.
  2. Disable parallel execution entirely by leaving that code but having it always behave as if _noparallel=1
  3. Continue digging and try and find some way to avoid this problem

The parallel execution work is something I was playing with last year in the hope of speeding up Datasette pages like the table page which need to execute a bunch of queries - one for each facet, plus one for each column to see if it should be suggested as a facet.

I wrote about this at the time here: https://simonwillison.net/2022/May/6/weeknotes/

My hope was that despite Python's GIL this optimization would still help, because the SQLite C module releases the GIL once it gets to SQLite.

But... that didn't hold up. It looked like enough work was happening in Python land with the GIL that the optimization didn't improve things.

Running the nogil fork of Python DID improve things though! I left the code in partly on the hope that the nogil fork would be accepted into Python core.

... which it now has! But it will still be a year or two before it fully lands: https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474

So I'm not particularly concerned about dropping the parallel execution. If I do drop it though do I leave the potentially complex code in that relates to it?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 1901416155