{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590681676", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590681676, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDY4MTY3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-25T04:48:29Z", "updated_at": "2020-02-25T04:48:29Z", "author_association": "OWNER", "body": "Documentation: https://datasette.readthedocs.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590679273", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590679273, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDY3OTI3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-25T04:37:21Z", "updated_at": "2020-02-25T04:37:21Z", "author_association": "OWNER", "body": "I'm happy with this now. I'm going to merge to master.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590617822", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590617822, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDYxNzgyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-25T00:26:48Z", "updated_at": "2020-02-25T00:26:48Z", "author_association": "OWNER", "body": "This failing test is a nasty one - the whole thing just hangs (so I imagine Travis will run for a while before hopefully giving up). Here's what happens if I add `--full-trace` and then hit Ctrl+C to cancel a test run:\r\n```\r\n$ pytest -k test_execute_write_fn_block_true --full-trace\r\n=================================================================== test session starts ===================================================================\r\nplatform darwin -- Python 3.7.5, pytest-5.2.4, py-1.8.1, pluggy-0.13.1\r\nrootdir: /Users/simonw/Dropbox/Development/datasette, inifile: pytest.ini\r\nplugins: asyncio-0.10.0\r\ncollected 410 items / 409 deselected / 1 selected                                                                                                         \r\n\r\ntests/test_database.py ^C^C\r\n\r\n================================================================= 409 deselected in 4.45s =================================================================\r\nTraceback (most recent call last):\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py\", line 193, in wrap_session\r\n    session.exitstatus = doit(config, session) or 0\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py\", line 237, in _main\r\n    config.hook.pytest_runtestloop(session=session)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py\", line 286, in __call__\r\n    return self._hookexec(self, self.get_hookimpls(), kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 93, in _hookexec\r\n    return self._inner_hookexec(hook, methods, kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 87, in <lambda>\r\n    firstresult=hook.spec.opts.get(\"firstresult\") if hook.spec else False,\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 208, in _multicall\r\n    return outcome.get_result()\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 80, in get_result\r\n    raise ex[1].with_traceback(ex[2])\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 187, in _multicall\r\n    res = hook_impl.function(*args)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py\", line 258, in pytest_runtestloop\r\n    item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py\", line 286, in __call__\r\n    return self._hookexec(self, self.get_hookimpls(), kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 93, in _hookexec\r\n    return self._inner_hookexec(hook, methods, kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 87, in <lambda>\r\n    firstresult=hook.spec.opts.get(\"firstresult\") if hook.spec else False,\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 208, in _multicall\r\n    return outcome.get_result()\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 80, in get_result\r\n    raise ex[1].with_traceback(ex[2])\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 187, in _multicall\r\n    res = hook_impl.function(*args)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py\", line 80, in pytest_runtest_protocol\r\n    runtestprotocol(item, nextitem=nextitem)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py\", line 95, in runtestprotocol\r\n    reports.append(call_and_report(item, \"call\", log))\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py\", line 176, in call_and_report\r\n    call = call_runtest_hook(item, when, **kwds)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py\", line 201, in call_runtest_hook\r\n    lambda: ihook(item=item, **kwds), when=when, reraise=reraise\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py\", line 229, in from_call\r\n    result = func()\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py\", line 201, in <lambda>\r\n    lambda: ihook(item=item, **kwds), when=when, reraise=reraise\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py\", line 286, in __call__\r\n    return self._hookexec(self, self.get_hookimpls(), kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 93, in _hookexec\r\n    return self._inner_hookexec(hook, methods, kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 87, in <lambda>\r\n    firstresult=hook.spec.opts.get(\"firstresult\") if hook.spec else False,\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 208, in _multicall\r\n    return outcome.get_result()\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 80, in get_result\r\n    raise ex[1].with_traceback(ex[2])\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 187, in _multicall\r\n    res = hook_impl.function(*args)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py\", line 125, in pytest_runtest_call\r\n    item.runtest()\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/python.py\", line 1429, in runtest\r\n    self.ihook.pytest_pyfunc_call(pyfuncitem=self)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py\", line 286, in __call__\r\n    return self._hookexec(self, self.get_hookimpls(), kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 93, in _hookexec\r\n    return self._inner_hookexec(hook, methods, kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 87, in <lambda>\r\n    firstresult=hook.spec.opts.get(\"firstresult\") if hook.spec else False,\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 208, in _multicall\r\n    return outcome.get_result()\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 80, in get_result\r\n    raise ex[1].with_traceback(ex[2])\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 187, in _multicall\r\n    res = hook_impl.function(*args)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pytest_asyncio/plugin.py\", line 158, in pytest_pyfunc_call\r\n    pyfuncitem.obj(**testargs), loop=event_loop))\r\n  File \"/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py\", line 566, in run_until_complete\r\n    self.run_forever()\r\n  File \"/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py\", line 534, in run_forever\r\n    self._run_once()\r\n  File \"/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py\", line 1735, in _run_once\r\n    event_list = self._selector.select(timeout)\r\n  File \"/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/selectors.py\", line 558, in select\r\n    kev_list = self._selector.control(None, max_ev, timeout)\r\nKeyboardInterrupt\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/bin/pytest\", line 8, in <module>\r\n    sys.exit(main())\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/config/__init__.py\", line 90, in main\r\n    return config.hook.pytest_cmdline_main(config=config)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py\", line 286, in __call__\r\n    return self._hookexec(self, self.get_hookimpls(), kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 93, in _hookexec\r\n    return self._inner_hookexec(hook, methods, kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 87, in <lambda>\r\n    firstresult=hook.spec.opts.get(\"firstresult\") if hook.spec else False,\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 208, in _multicall\r\n    return outcome.get_result()\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 80, in get_result\r\n    raise ex[1].with_traceback(ex[2])\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 187, in _multicall\r\n    res = hook_impl.function(*args)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py\", line 230, in pytest_cmdline_main\r\n    return wrap_session(config, _main)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py\", line 209, in wrap_session\r\n    config.hook.pytest_keyboard_interrupt(excinfo=excinfo)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py\", line 286, in __call__\r\n    return self._hookexec(self, self.get_hookimpls(), kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 93, in _hookexec\r\n    return self._inner_hookexec(hook, methods, kwargs)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py\", line 87, in <lambda>\r\n    firstresult=hook.spec.opts.get(\"firstresult\") if hook.spec else False,\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 208, in _multicall\r\n    return outcome.get_result()\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 80, in get_result\r\n    raise ex[1].with_traceback(ex[2])\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py\", line 187, in _multicall\r\n    res = hook_impl.function(*args)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/terminal.py\", line 680, in pytest_keyboard_interrupt\r\n    self._keyboardinterrupt_memo = excinfo.getrepr(funcargs=True)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py\", line 598, in getrepr\r\n    return fmt.repr_excinfo(self)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py\", line 830, in repr_excinfo\r\n    reprtraceback = self.repr_traceback(excinfo)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py\", line 778, in repr_traceback\r\n    reprentry = self.repr_traceback_entry(entry, einfo)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py\", line 737, in repr_traceback_entry\r\n    reprargs = self.repr_args(entry) if not short else None\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py\", line 656, in repr_args\r\n    args.append((argname, saferepr(argvalue)))\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py\", line 67, in saferepr\r\n    return SafeRepr(maxsize).repr(obj)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py\", line 36, in repr\r\n    s = super().repr(x)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py\", line 52, in repr\r\n    return self.repr1(x, self.maxlevel)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py\", line 60, in repr1\r\n    return getattr(self, 'repr_' + typename)(x, level)\r\n  File \"/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py\", line 112, in repr_dict\r\n    for key in islice(_possibly_sorted(x), self.maxdict):\r\nKeyboardInterrupt\r\nTask was destroyed but it is pending!\r\ntask: <Task pending coro=<test_execute_write_fn_block_true() running at /Users/simonw/Dropbox/Development/datasette/tests/test_database.py:139> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x105580510>()]>>\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590614896", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590614896, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDYxNDg5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-25T00:16:51Z", "updated_at": "2020-02-25T00:16:51Z", "author_association": "OWNER", "body": "The other problem with the poll-for-UUID-completion idea: how long does this mean Datasette needs to keep holding onto the `WriteTask` objects?\r\n\r\nMaybe we say you only get to ask \"is this UUID still in the queue\" and if the answer is \"no\" then you assume the task has been completed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590610180", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590610180, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDYxMDE4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-25T00:00:07Z", "updated_at": "2020-02-25T00:00:07Z", "author_association": "OWNER", "body": "Basic stuff to cover in unit tests:\r\n- Exercise `.execute_write(sql)` - both with block=True and block=False\r\n- Exercise `.execute_write_fn(fn)` in the same way\r\n- Throw 10 updates in the queue, block on just the last one, check it worked correctly\r\n\r\nI'm going to write these tests directly against a `Database()` object rather than booting up an entire Datasette instance.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590608228", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590608228, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDYwODIyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:52:35Z", "updated_at": "2020-02-24T23:52:35Z", "author_association": "OWNER", "body": "I'm going to punt on the ability to introspect the write queue and poll for completion using a UUID for the moment. Can add those later.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590607385", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590607385, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDYwNzM4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:49:37Z", "updated_at": "2020-02-24T23:49:37Z", "author_association": "OWNER", "body": "Here's the `upload_csv.py` plugin file I've been playing with:\r\n```python\r\nfrom datasette import hookimpl\r\nfrom starlette.responses import PlainTextResponse, HTMLResponse\r\nfrom starlette.endpoints import HTTPEndpoint\r\nimport csv as csv_std\r\nimport codecs\r\nimport sqlite_utils\r\n\r\n\r\nclass UploadApp(HTTPEndpoint):\r\n    def __init__(self, scope, receive, send, datasette):\r\n        self.datasette = datasette\r\n        super().__init__(scope, receive, send)\r\n\r\n    def get_database(self):\r\n        # For the moment just use the first one that's not immutable\r\n        mutable = [db for db in self.datasette.databases.values() if db.is_mutable]\r\n        return mutable[0]\r\n\r\n    async def get(self, request):\r\n        return HTMLResponse(\r\n            await self.datasette.render_template(\r\n                \"upload_csv.html\", {\"database_name\": self.get_database().name}\r\n            )\r\n        )\r\n\r\n    async def post(self, request):\r\n        formdata = await request.form()\r\n        csv = formdata[\"csv\"]\r\n        # csv.file is a SpooledTemporaryFile, I can read it directly\r\n        filename = csv.filename\r\n        # TODO: Support other encodings:\r\n        reader = csv_std.reader(codecs.iterdecode(csv.file, \"utf-8\"))\r\n        headers = next(reader)\r\n        docs = (dict(zip(headers, row)) for row in reader)\r\n        if filename.endswith(\".csv\"):\r\n            filename = filename[:-4]\r\n        # Import data into a table of that name using sqlite-utils\r\n        db = self.get_database()\r\n\r\n        def fn(conn):\r\n            writable_conn = sqlite_utils.Database(db.path)\r\n            writable_conn[filename].insert_all(docs, alter=True)\r\n            return writable_conn[filename].count\r\n\r\n        # Without block=True we may attempt 'select count(*) from ...'\r\n        # before the table has been created by the write thread\r\n        count = await db.execute_write_fn(fn, block=True)\r\n\r\n        return HTMLResponse(\r\n            await self.datasette.render_template(\r\n                \"upload_csv_done.html\",\r\n                {\r\n                    \"database\": self.get_database().name,\r\n                    \"table\": filename,\r\n                    \"num_docs\": count,\r\n                },\r\n            )\r\n        )\r\n\r\n\r\n@hookimpl\r\ndef asgi_wrapper(datasette):\r\n    def wrap_with_asgi_auth(app):\r\n        async def wrapped_app(scope, recieve, send):\r\n            if scope[\"path\"] == \"/-/upload-csv\":\r\n                await UploadApp(scope, recieve, send, datasette)\r\n            else:\r\n                await app(scope, recieve, send)\r\n\r\n        return wrapped_app\r\n\r\n    return wrap_with_asgi_auth\r\n```\r\nI also dropped copies of the two template files from https://github.com/simonw/datasette-upload-csvs/tree/699e6ca591f36264bfc8e590d877e6852f274beb/datasette_upload_csvs/templates into my `write-templates/` directory.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590606825", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590606825, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDYwNjgyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:47:38Z", "updated_at": "2020-02-24T23:47:38Z", "author_association": "OWNER", "body": "Another demo plugin: `delete_table.py`\r\n```python\r\nfrom datasette import hookimpl\r\nfrom datasette.utils import escape_sqlite\r\nfrom starlette.responses import HTMLResponse\r\nfrom starlette.endpoints import HTTPEndpoint\r\n\r\n\r\nclass DeleteTableApp(HTTPEndpoint):\r\n    def __init__(self, scope, receive, send, datasette):\r\n        self.datasette = datasette\r\n        super().__init__(scope, receive, send)\r\n\r\n    async def post(self, request):\r\n        formdata = await request.form()\r\n        database = formdata[\"database\"]\r\n        db = self.datasette.databases[database]\r\n        await db.execute_write(\"drop table {}\".format(escape_sqlite(formdata[\"table\"])))\r\n        return HTMLResponse(\"Table has been deleted.\")\r\n\r\n\r\n@hookimpl\r\ndef asgi_wrapper(datasette):\r\n    def wrap_with_asgi_auth(app):\r\n        async def wrapped_app(scope, recieve, send):\r\n            if scope[\"path\"] == \"/-/delete-table\":\r\n                await DeleteTableApp(scope, recieve, send, datasette)\r\n            else:\r\n                await app(scope, recieve, send)\r\n\r\n        return wrapped_app\r\n\r\n    return wrap_with_asgi_auth\r\n```\r\nThen I saved this as `table.html` in the `write-templates/` directory:\r\n```html+django\r\n{% extends \"default:table.html\" %}\r\n\r\n{% block content %}\r\n<form action=\"/-/delete-table\" method=\"POST\">\r\n    <p>\r\n        <input type=\"hidden\" name=\"database\" value=\"{{ database }}\">\r\n        <input type=\"hidden\" name=\"table\" value=\"{{ table }}\">\r\n        <input type=\"submit\" value=\"Delete this table\">\r\n    </p>\r\n</form>\r\n{{ super() }}\r\n{% endblock %}\r\n```\r\n(Needs CSRF protection added)\r\n\r\nI ran Datasette like this:\r\n\r\n    $ datasette --plugins-dir=write-plugins/ data.db --template-dir=write-templates/\r\n\r\nResult: I can delete tables!\r\n\r\n<img width=\"596\" alt=\"data__everything__30_132_rows_-_Mozilla_Firefox\" src=\"https://user-images.githubusercontent.com/9599/75201302-f9cec580-571c-11ea-9c55-67a49e68ec0c.png\">\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590599257", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590599257, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDU5OTI1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:21:56Z", "updated_at": "2020-02-24T23:22:35Z", "author_association": "OWNER", "body": "Also: are UUIDs really necessary here or could I use a simpler form of task identifier? Like an in-memory counter variable that starts at 0 and increments every time this instance of Datasette issues a new task ID?\r\n\r\nThe neat thing about UUIDs is that I don't have to worry if there are multiple Datasette instances accepting writes behind a load balancer. That seems pretty unlikely (especially considering SQLite databases encourage only one process to be writing at a time)... but I am experimenting with PostgreSQL support in #670 so it's probably worth ensuring these task IDs really are globally unique.\r\n\r\nI'm going to stick with UUIDs. They're short-lived enough that their size doesn't really matter.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590598689", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590598689, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDU5ODY4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:20:11Z", "updated_at": "2020-02-24T23:20:11Z", "author_association": "OWNER", "body": "I think `if block` it makes sense to return the return value of the function that was executed. Without it all I really need to do is return the `uuid` so something could theoretically poll for completion later on.\r\n\r\nBut is it weird having a function that returns different types depending on if you passed `block=True` or not? Should they be differently named functions?\r\n\r\nI'm OK with the `block=True` pattern changing the return value I think.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590598248", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590598248, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDU5ODI0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:18:50Z", "updated_at": "2020-02-24T23:18:50Z", "author_association": "OWNER", "body": "I'm not convinced by the return value of the `.execute_write_fn()` method:\r\n\r\nhttps://github.com/simonw/datasette/blob/ab2348280206bde1390b931ae89d372c2f74b87e/datasette/database.py#L79-L83\r\n\r\nDo I really need that `WriteResponse` class or can I do something nicer?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590593120", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590593120, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDU5MzEyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:02:30Z", "updated_at": "2020-02-24T23:02:30Z", "author_association": "OWNER", "body": "I'm going to muck around with a couple more demo plugins - in particular one derived from [datasette-upload-csvs](https://github.com/simonw/datasette-upload-csvs) - to make sure I'm comfortable with this API - then add a couple of tests and merge it with documentation that warns \"this is still an experimental feature and may change\".", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590592581", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590592581, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDU5MjU4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:00:44Z", "updated_at": "2020-02-24T23:01:09Z", "author_association": "OWNER", "body": "I've been testing this out by running one-off demo plugins. I saved the following in a file called `write-plugins/log_asgi.py` (it's a hacked around copy of [asgi-log-to-sqlite](https://github.com/simonw/asgi-log-to-sqlite)) and then running `datasette data.db --plugins-dir=write-plugins/`:\r\n```python\r\nfrom datasette import hookimpl\r\nimport sqlite_utils\r\nimport time\r\n\r\n\r\nclass AsgiLogToSqliteViaWriteQueue:\r\n    lookup_columns = (\r\n        \"path\",\r\n        \"user_agent\",\r\n        \"referer\",\r\n        \"accept_language\",\r\n        \"content_type\",\r\n        \"query_string\",\r\n    )\r\n\r\n    def __init__(self, app, db):\r\n        self.app = app\r\n        self.db = db\r\n        self._tables_ensured = False\r\n\r\n    async def ensure_tables(self):\r\n        def _ensure_tables(conn):\r\n            db = sqlite_utils.Database(conn)\r\n            for column in self.lookup_columns:\r\n                table = \"{}s\".format(column)\r\n                if not db[table].exists():\r\n                    db[table].create({\"id\": int, \"name\": str}, pk=\"id\")\r\n            if \"requests\" not in db.table_names():\r\n                db[\"requests\"].create(\r\n                    {\r\n                        \"start\": float,\r\n                        \"method\": str,\r\n                        \"path\": int,\r\n                        \"query_string\": int,\r\n                        \"user_agent\": int,\r\n                        \"referer\": int,\r\n                        \"accept_language\": int,\r\n                        \"http_status\": int,\r\n                        \"content_type\": int,\r\n                        \"client_ip\": str,\r\n                        \"duration\": float,\r\n                        \"body_size\": int,\r\n                    },\r\n                    foreign_keys=self.lookup_columns,\r\n                )\r\n        await self.db.execute_write_fn(_ensure_tables)\r\n\r\n    async def __call__(self, scope, receive, send):\r\n        if not self._tables_ensured:\r\n            self._tables_ensured = True\r\n            await self.ensure_tables()\r\n\r\n        response_headers = []\r\n        body_size = 0\r\n        http_status = None\r\n\r\n        async def wrapped_send(message):\r\n            nonlocal body_size, response_headers, http_status\r\n            if message[\"type\"] == \"http.response.start\":\r\n                response_headers = message[\"headers\"]\r\n                http_status = message[\"status\"]\r\n\r\n            if message[\"type\"] == \"http.response.body\":\r\n                body_size += len(message[\"body\"])\r\n\r\n            await send(message)\r\n\r\n        start = time.time()\r\n        await self.app(scope, receive, wrapped_send)\r\n        end = time.time()\r\n\r\n        path = str(scope[\"path\"])\r\n        query_string = None\r\n        if scope.get(\"query_string\"):\r\n            query_string = \"?{}\".format(scope[\"query_string\"].decode(\"utf8\"))\r\n\r\n        request_headers = dict(scope.get(\"headers\") or [])\r\n\r\n        referer = header(request_headers, \"referer\")\r\n        user_agent = header(request_headers, \"user-agent\")\r\n        accept_language = header(request_headers, \"accept-language\")\r\n\r\n        content_type = header(dict(response_headers), \"content-type\")\r\n\r\n        def _log_to_database(conn):\r\n            db = sqlite_utils.Database(conn)\r\n            db[\"requests\"].insert(\r\n                {\r\n                    \"start\": start,\r\n                    \"method\": scope[\"method\"],\r\n                    \"path\": lookup(db, \"paths\", path),\r\n                    \"query_string\": lookup(db, \"query_strings\", query_string),\r\n                    \"user_agent\": lookup(db, \"user_agents\", user_agent),\r\n                    \"referer\": lookup(db, \"referers\", referer),\r\n                    \"accept_language\": lookup(db, \"accept_languages\", accept_language),\r\n                    \"http_status\": http_status,\r\n                    \"content_type\": lookup(db, \"content_types\", content_type),\r\n                    \"client_ip\": scope.get(\"client\", (None, None))[0],\r\n                    \"duration\": end - start,\r\n                    \"body_size\": body_size,\r\n                },\r\n                alter=True,\r\n                foreign_keys=self.lookup_columns,\r\n            )\r\n\r\n        await self.db.execute_write_fn(_log_to_database)\r\n\r\n\r\ndef header(d, name):\r\n    return d.get(name.encode(\"utf8\"), b\"\").decode(\"utf8\") or None\r\n\r\n\r\ndef lookup(db, table, value):\r\n    return db[table].lookup({\"name\": value}) if value else None\r\n\r\n\r\n@hookimpl\r\ndef asgi_wrapper(datasette):\r\n    def wrap_with_class(app):\r\n        return AsgiLogToSqliteViaWriteQueue(\r\n            app, next(iter(datasette.databases.values()))\r\n        )\r\n\r\n    return wrap_with_class\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590518182", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590518182, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDUxODE4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T19:53:12Z", "updated_at": "2020-02-24T19:53:12Z", "author_association": "OWNER", "body": "Next steps are from comment https://github.com/simonw/datasette/issues/682#issuecomment-590517338\r\n> I'm going to move ahead without needing that ability though. I figure SQLite writes are _fast_, and plugins can be trusted to implement just fast writes. So I'm going to support either fire-and-forget writes (they get added to the queue and a task ID is returned) or have the option to block awaiting the completion of the write (using Janus) but let callers decide which version they want. I may add optional timeouts some time in the future.\r\n> \r\n> I am going to make both `execute_write()` and `execute_write_fn()` awaitable functions though, for consistency with `.execute()` and to give me flexibility to change how they work in the future.\r\n> \r\n> I'll also add a `block=True` option to both of them which causes the function to wait for the write to be successfully executed - defaults to `False` (fire-and-forget mode).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null}