{"html_url": "https://github.com/simonw/datasette/issues/682#issuecomment-590517744", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/682", "id": 590517744, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDUxNzc0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T19:52:16Z", "updated_at": "2020-02-24T19:52:16Z", "author_association": "OWNER", "body": "Moving further development to a pull request: https://github.com/simonw/datasette/pull/683", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 569613563, "label": "Mechanism for writing to database via a queue"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/682#issuecomment-590517338", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/682", "id": 590517338, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDUxNzMzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T19:51:21Z", "updated_at": "2020-02-24T19:51:21Z", "author_association": "OWNER", "body": "I filed a question / feature request with Janus about supporting timeouts for `.get()` against async queues here: https://github.com/aio-libs/janus/issues/240\r\n\r\nI'm going to move ahead without needing that ability though. I figure SQLite writes are _fast_, and plugins can be trusted to implement just fast writes. So I'm going to support either fire-and-forget writes (they get added to the queue and a task ID is returned) or have the option to block awaiting the completion of the write (using Janus) but let callers decide which version they want. I may add optional timeouts some time in the future.\r\n\r\nI am going to make both `execute_write()` and `execute_write_fn()` awaitable functions though, for consistency with `.execute()` and to give me flexibility to change how they work in the future.\r\n\r\nI'll also add a `block=True` option to both of them which causes the function to wait for the write to be successfully executed - defaults to `False` (fire-and-forget mode).\r\n", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 569613563, "label": "Mechanism for writing to database via a queue"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/682#issuecomment-590511601", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/682", "id": 590511601, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDUxMTYwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T19:38:27Z", "updated_at": "2020-02-24T19:38:27Z", "author_association": "OWNER", "body": "I tested this using the following code in a view (after `from sqlite_utils import Database`):\r\n```python\r\n    db = next(iter(self.ds.databases.values()))\r\n    db.execute_write_fn(lambda conn: Database(conn)[\"counter\"].insert({\"id\": 1, \"count\": 0}, pk=\"id\", ignore=True))\r\n    db.execute_write(\"update counter set count = count + 1 where id = 1\")\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 569613563, "label": "Mechanism for writing to database via a queue"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/682#issuecomment-590436368", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/682", "id": 590436368, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDQzNjM2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T17:00:21Z", "updated_at": "2020-02-24T17:00:21Z", "author_association": "OWNER", "body": "Interesting challenge: I would like to be able to \"await\" on `queue.get()` (with a timeout).\r\n\r\nProblem is: `queue.Queue()` is designed for threading and cannot be awaited. `asyncio.Queue` can be awaited but is not meant to be used with threads.\r\n\r\nhttps://stackoverflow.com/a/32894169 suggests using Janus, a thread-aware asyncio queue: https://github.com/aio-libs/janus", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 569613563, "label": "Mechanism for writing to database via a queue"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/682#issuecomment-590430988", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/682", "id": 590430988, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDQzMDk4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T16:50:48Z", "updated_at": "2020-02-24T16:50:48Z", "author_association": "OWNER", "body": "I'm dropping the progress bar idea. This mechanism is supposed to guarantee exclusive access to the single write connection, which means it should be targeted by operations that are as short as possible. An operation running long enough to need a progress bar is too long!\r\n\r\nAny implementation of progress bars for long running write operations needs to happen elsewhere in the stack.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 569613563, "label": "Mechanism for writing to database via a queue"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/682#issuecomment-590417619", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/682", "id": 590417619, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDQxNzYxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T16:27:36Z", "updated_at": "2020-02-24T16:27:36Z", "author_association": "OWNER", "body": "Error handling could be tricky. Exceptions thrown in threads don't show up anywhere by default - I would need to explicitly catch them and decide what to do with them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 569613563, "label": "Mechanism for writing to database via a queue"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/682#issuecomment-590417366", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/682", "id": 590417366, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDQxNzM2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T16:27:10Z", "updated_at": "2020-02-24T16:27:10Z", "author_association": "OWNER", "body": "I wonder if I even need the `reply_queue` mechanism? Are the replies from writes generally even interesting?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 569613563, "label": "Mechanism for writing to database via a queue"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/682#issuecomment-590399600", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/682", "id": 590399600, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDM5OTYwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T15:56:10Z", "updated_at": "2020-02-24T15:56:23Z", "author_association": "OWNER", "body": "## Implementation plan\r\n\r\nMethod on Database class called `execute_write(sql)`\r\n\r\nWhich calls `.execute_write_fn(fn)` - so you can instead create a function that applies a whole batch of writes and pass that instead if you need to\r\n\r\nThrows an error of database isn't mutable.\r\n\r\nAdd `._writer_thread` thread property to Database - we start that thread the first time we need it. It blocks on `._writer_queue.get()`\r\n\r\nWe write to that queue with `WriteTask(fn, uuid, reply_queue)` namedtuples - then time-out block awaiting reply for 0.5s\r\n\r\nHave a `.write_status(uuid)` method that checks if `uuid` has completed\r\n\r\nThis should be enough to get it all working. MVP can skip the .5s timeout entirely\r\n\r\nBut... what about that progress bar supporting stretch goal?\r\n\r\nFor that let's have each write operation that's currently in progress have total and done integer properties. So I guess we can add those to the `WriteTask`.\r\n\r\nShould we have the ability to see what the currently executing write is? Seems useful.\r\n\r\nHopefully I can integrate https://github.com/tqdm/tqdm such that it calculates ETAs without actually trying to print to the console.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 569613563, "label": "Mechanism for writing to database via a queue"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/682#issuecomment-590154309", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/682", "id": 590154309, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDE1NDMwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T03:14:10Z", "updated_at": "2020-02-24T03:14:10Z", "author_association": "OWNER", "body": "Some prior art: Charles Leifer implemented a `SqliteQueueDatabase` class that automatically queues writes for you: https://charlesleifer.com/blog/multi-threaded-sqlite-without-the-operationalerrors/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 569613563, "label": "Mechanism for writing to database via a queue"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/682#issuecomment-590153892", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/682", "id": 590153892, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDE1Mzg5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T03:10:45Z", "updated_at": "2020-02-24T03:13:03Z", "author_association": "OWNER", "body": "Some more detailed notes I made earlier:\r\n\r\nDatasette would run a single write thread per database. That thread gets an exclusive connection, and a queue. Plugins can add functions to the queue which will be called and given access to that connection.\r\n\r\nThe write thread for that database is created the first time a write is attempted.\r\n\r\nQuestion: should that thread have its own asyncio loop so that async techniques like httpx can be used within the thread? I think not at first - only investigate this if it turns out to be necessary in the future.\r\n\r\nThis thread will run as part of the Datasette process. This means there is always a risk that the thread will die in the middle of something because the server got restarted - so use transactions to limit risk of damage to database should that happen.\r\n\r\nI don\u2019t want web responses blocking waiting for stuff to happen here - so every task put on that queue will have a task ID, and that ID will be returned such that client code can poll for its completion.\r\n\r\nCould the request block for up to 0.5s just in case the write is really fast, then return a polling token if it isn't finished yet? Looks possible - `Queue.get` can block with a timeout.\r\n\r\nThere will be a `/-/writes` page which shows currently queued writes - so each one needs a human-readable description of some sort. (You can access a deque called `q.queue` to see what\u2019s in there)\r\n\r\nStretch goal: It would be cool if write operations could optionally handle their own progress reports. That way I can do some really nice UI around what\u2019s going on with these things.\r\n\r\nThis mechanism has a ton of potential. It may even be how we handle things like Twitter imports and suchlike - queued writing tasks.\r\n\r\nOne catch with this approach: if a plugin is reading from APIs etc it shouldn't block writes to the database while it is doing so. So sticking a function in the queue that does additional time consuming stuff is actually an anti pattern. Instead, plugins should schedule their API access in the main event loop and occasionally write just the updates they need to make to that write queue.\r\n\r\n### Implementation notes\r\n\r\nMaybe each item in the queue is a `(callable, uuid, reply_queue)` triple. You can do a blocking `.get()` on the `reply_queue` if you want to wait for the answer. The execution framework could look for the return value from `callable()` and automatically send it to `reply_queue`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 569613563, "label": "Mechanism for writing to database via a queue"}, "performed_via_github_app": null}