issue_comments

Documentation: https://datasette.readthedocs.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false

I'm happy with this now. I'm going to merge to master.

This failing test is a nasty one - the whole thing just hangs (so I imagine Travis will run for a while before hopefully giving up). Here's what happens if I add --full-trace and then hit Ctrl+C to cancel a test run: ``` $ pytest -k test_execute_write_fn_block_true --full-trace =================================================================== test session starts =================================================================== platform darwin -- Python 3.7.5, pytest-5.2.4, py-1.8.1, pluggy-0.13.1 rootdir: /Users/simonw/Dropbox/Development/datasette, inifile: pytest.ini plugins: asyncio-0.10.0 collected 410 items / 409 deselected / 1 selected

tests/test_database.py ^C^C

================================================================= 409 deselected in 4.45s ================================================================= Traceback (most recent call last): File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 193, in wrap_session session.exitstatus = doit(config, session) or 0 File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 237, in _main config.hook.pytest_runtestloop(session=session) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in call return self._hookexec(self, self.get_hookimpls(), kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall return outcome.get_result() File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(args) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 258, in pytest_runtestloop item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in call return self._hookexec(self, self.get_hookimpls(), kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall return outcome.get_result() File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(args) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 80, in pytest_runtest_protocol runtestprotocol(item, nextitem=nextitem) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 95, in runtestprotocol reports.append(call_and_report(item, "call", log)) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 176, in call_and_report call = call_runtest_hook(item, when, kwds) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 201, in call_runtest_hook lambda: ihook(item=item, kwds), when=when, reraise=reraise File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 229, in from_call result = func() File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 201, in <lambda> lambda: ihook(item=item, kwds), when=when, reraise=reraise File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in call return self._hookexec(self, self.get_hookimpls(), kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall return outcome.get_result() File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(args) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 125, in pytest_runtest_call item.runtest() File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/python.py", line 1429, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in call return self._hookexec(self, self.get_hookimpls(), kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall return outcome.get_result() File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(args) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pytest_asyncio/plugin.py", line 158, in pytest_pyfunc_call pyfuncitem.obj(testargs), loop=event_loop)) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 566, in run_until_complete self.run_forever() File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 534, in run_forever self._run_once() File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 1735, in _run_once event_list = self._selector.select(timeout) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/selectors.py", line 558, in select kev_list = self._selector.control(None, max_ev, timeout) KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/bin/pytest", line 8, in <module> sys.exit(main()) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pytest/config/__init__.py", line 90, in main return config.hook.pytest_cmdline_main(config=config) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__ return self._hookexec(self, self.get_hookimpls(), kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall return outcome.get_result() File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(args) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 230, in pytest_cmdline_main return wrap_session(config, _main) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 209, in wrap_session config.hook.pytest_keyboard_interrupt(excinfo=excinfo) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in call return self._hookexec(self, self.get_hookimpls(), kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall return outcome.get_result() File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(args) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/terminal.py", line 680, in pytest_keyboard_interrupt self._keyboardinterrupt_memo = excinfo.getrepr(funcargs=True) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 598, in getrepr return fmt.repr_excinfo(self) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 830, in repr_excinfo reprtraceback = self.repr_traceback(excinfo) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 778, in repr_traceback reprentry = self.repr_traceback_entry(entry, einfo) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 737, in repr_traceback_entry reprargs = self.repr_args(entry) if not short else None File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 656, in repr_args args.append((argname, saferepr(argvalue))) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py", line 67, in saferepr return SafeRepr(maxsize).repr(obj) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py", line 36, in repr s = super().repr(x) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 52, in repr return self.repr1(x, self.maxlevel) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 60, in repr1 return getattr(self, 'repr' + typename)(x, level) File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 112, in repr_dict for key in islice(_possibly_sorted(x), self.maxdict): KeyboardInterrupt Task was destroyed but it is pending! task: <Task pending coro=\<test_execute_write_fn_block_true() running at /Users/simonw/Dropbox/Development/datasette/tests/test_database.py:139> wait_for=\<Future pending cb=[\<TaskWakeupMethWrapper object at 0x105580510>()]>> ```

The other problem with the poll-for-UUID-completion idea: how long does this mean Datasette needs to keep holding onto the WriteTask objects?

Maybe we say you only get to ask "is this UUID still in the queue" and if the answer is "no" then you assume the task has been completed.

Basic stuff to cover in unit tests: - Exercise .execute_write(sql) - both with block=True and block=False - Exercise .execute_write_fn(fn) in the same way - Throw 10 updates in the queue, block on just the last one, check it worked correctly

I'm going to write these tests directly against a Database() object rather than booting up an entire Datasette instance.

I'm going to punt on the ability to introspect the write queue and poll for completion using a UUID for the moment. Can add those later.

Here's the upload_csv.py plugin file I've been playing with: ```python from datasette import hookimpl from starlette.responses import PlainTextResponse, HTMLResponse from starlette.endpoints import HTTPEndpoint import csv as csv_std import codecs import sqlite_utils

class UploadApp(HTTPEndpoint): def init(self, scope, receive, send, datasette): self.datasette = datasette super().init(scope, receive, send)

def get_database(self):
    # For the moment just use the first one that's not immutable
    mutable = [db for db in self.datasette.databases.values() if db.is_mutable]
    return mutable[0]

async def get(self, request):
    return HTMLResponse(
        await self.datasette.render_template(
            "upload_csv.html", {"database_name": self.get_database().name}
        )
    )

async def post(self, request):
    formdata = await request.form()
    csv = formdata["csv"]
    # csv.file is a SpooledTemporaryFile, I can read it directly
    filename = csv.filename
    # TODO: Support other encodings:
    reader = csv_std.reader(codecs.iterdecode(csv.file, "utf-8"))
    headers = next(reader)
    docs = (dict(zip(headers, row)) for row in reader)
    if filename.endswith(".csv"):
        filename = filename[:-4]
    # Import data into a table of that name using sqlite-utils
    db = self.get_database()

    def fn(conn):
        writable_conn = sqlite_utils.Database(db.path)
        writable_conn[filename].insert_all(docs, alter=True)
        return writable_conn[filename].count

    # Without block=True we may attempt 'select count(*) from ...'
    # before the table has been created by the write thread
    count = await db.execute_write_fn(fn, block=True)

    return HTMLResponse(
        await self.datasette.render_template(
            "upload_csv_done.html",
            {
                "database": self.get_database().name,
                "table": filename,
                "num_docs": count,
            },
        )
    )

@hookimpl def asgi_wrapper(datasette): def wrap_with_asgi_auth(app): async def wrapped_app(scope, recieve, send): if scope["path"] == "/-/upload-csv": await UploadApp(scope, recieve, send, datasette) else: await app(scope, recieve, send)

    return wrapped_app

return wrap_with_asgi_auth

`` I also dropped copies of the two template files from https://github.com/simonw/datasette-upload-csvs/tree/699e6ca591f36264bfc8e590d877e6852f274beb/datasette_upload_csvs/templates into mywrite-templates/` directory.

Another demo plugin: delete_table.py ```python from datasette import hookimpl from datasette.utils import escape_sqlite from starlette.responses import HTMLResponse from starlette.endpoints import HTTPEndpoint

class DeleteTableApp(HTTPEndpoint): def init(self, scope, receive, send, datasette): self.datasette = datasette super().init(scope, receive, send)

async def post(self, request):
    formdata = await request.form()
    database = formdata["database"]
    db = self.datasette.databases[database]
    await db.execute_write("drop table {}".format(escape_sqlite(formdata["table"])))
    return HTMLResponse("Table has been deleted.")

@hookimpl def asgi_wrapper(datasette): def wrap_with_asgi_auth(app): async def wrapped_app(scope, recieve, send): if scope["path"] == "/-/delete-table": await DeleteTableApp(scope, recieve, send, datasette) else: await app(scope, recieve, send)

    return wrapped_app

return wrap_with_asgi_auth

Then I saved this as `table.html` in the `write-templates/` directory:html+django {% extends "default:table.html" %}

{% block content %}

{{ super() }} {% endblock %} ``` (Needs CSRF protection added)

I ran Datasette like this:

$ datasette --plugins-dir=write-plugins/ data.db --template-dir=write-templates/

Result: I can delete tables!

Also: are UUIDs really necessary here or could I use a simpler form of task identifier? Like an in-memory counter variable that starts at 0 and increments every time this instance of Datasette issues a new task ID?

The neat thing about UUIDs is that I don't have to worry if there are multiple Datasette instances accepting writes behind a load balancer. That seems pretty unlikely (especially considering SQLite databases encourage only one process to be writing at a time)... but I am experimenting with PostgreSQL support in #670 so it's probably worth ensuring these task IDs really are globally unique.

I'm going to stick with UUIDs. They're short-lived enough that their size doesn't really matter.

I think if block it makes sense to return the return value of the function that was executed. Without it all I really need to do is return the uuid so something could theoretically poll for completion later on.

But is it weird having a function that returns different types depending on if you passed block=True or not? Should they be differently named functions?

I'm OK with the block=True pattern changing the return value I think.

I'm not convinced by the return value of the .execute_write_fn() method:

https://github.com/simonw/datasette/blob/ab2348280206bde1390b931ae89d372c2f74b87e/datasette/database.py#L79-L83

Do I really need that WriteResponse class or can I do something nicer?

I'm going to muck around with a couple more demo plugins - in particular one derived from datasette-upload-csvs - to make sure I'm comfortable with this API - then add a couple of tests and merge it with documentation that warns "this is still an experimental feature and may change".

I've been testing this out by running one-off demo plugins. I saved the following in a file called write-plugins/log_asgi.py (it's a hacked around copy of asgi-log-to-sqlite) and then running datasette data.db --plugins-dir=write-plugins/: ```python from datasette import hookimpl import sqlite_utils import time

class AsgiLogToSqliteViaWriteQueue: lookup_columns = ( "path", "user_agent", "referer", "accept_language", "content_type", "query_string", )

def __init__(self, app, db):
    self.app = app
    self.db = db
    self._tables_ensured = False

async def ensure_tables(self):
    def _ensure_tables(conn):
        db = sqlite_utils.Database(conn)
        for column in self.lookup_columns:
            table = "{}s".format(column)
            if not db[table].exists():
                db[table].create({"id": int, "name": str}, pk="id")
        if "requests" not in db.table_names():
            db["requests"].create(
                {
                    "start": float,
                    "method": str,
                    "path": int,
                    "query_string": int,
                    "user_agent": int,
                    "referer": int,
                    "accept_language": int,
                    "http_status": int,
                    "content_type": int,
                    "client_ip": str,
                    "duration": float,
                    "body_size": int,
                },
                foreign_keys=self.lookup_columns,
            )
    await self.db.execute_write_fn(_ensure_tables)

async def __call__(self, scope, receive, send):
    if not self._tables_ensured:
        self._tables_ensured = True
        await self.ensure_tables()

    response_headers = []
    body_size = 0
    http_status = None

    async def wrapped_send(message):
        nonlocal body_size, response_headers, http_status
        if message["type"] == "http.response.start":
            response_headers = message["headers"]
            http_status = message["status"]

        if message["type"] == "http.response.body":
            body_size += len(message["body"])

        await send(message)

    start = time.time()
    await self.app(scope, receive, wrapped_send)
    end = time.time()

    path = str(scope["path"])
    query_string = None
    if scope.get("query_string"):
        query_string = "?{}".format(scope["query_string"].decode("utf8"))

    request_headers = dict(scope.get("headers") or [])

    referer = header(request_headers, "referer")
    user_agent = header(request_headers, "user-agent")
    accept_language = header(request_headers, "accept-language")

    content_type = header(dict(response_headers), "content-type")

    def _log_to_database(conn):
        db = sqlite_utils.Database(conn)
        db["requests"].insert(
            {
                "start": start,
                "method": scope["method"],
                "path": lookup(db, "paths", path),
                "query_string": lookup(db, "query_strings", query_string),
                "user_agent": lookup(db, "user_agents", user_agent),
                "referer": lookup(db, "referers", referer),
                "accept_language": lookup(db, "accept_languages", accept_language),
                "http_status": http_status,
                "content_type": lookup(db, "content_types", content_type),
                "client_ip": scope.get("client", (None, None))[0],
                "duration": end - start,
                "body_size": body_size,
            },
            alter=True,
            foreign_keys=self.lookup_columns,
        )

    await self.db.execute_write_fn(_log_to_database)

def header(d, name): return d.get(name.encode("utf8"), b"").decode("utf8") or None

def lookup(db, table, value): return db[table].lookup({"name": value}) if value else None

@hookimpl def asgi_wrapper(datasette): def wrap_with_class(app): return AsgiLogToSqliteViaWriteQueue( app, next(iter(datasette.databases.values())) )

return wrap_with_class

```

Next steps are from comment https://github.com/simonw/datasette/issues/682#issuecomment-590517338

I'm going to move ahead without needing that ability though. I figure SQLite writes are fast, and plugins can be trusted to implement just fast writes. So I'm going to support either fire-and-forget writes (they get added to the queue and a task ID is returned) or have the option to block awaiting the completion of the write (using Janus) but let callers decide which version they want. I may add optional timeouts some time in the future.

I am going to make both execute_write() and execute_write_fn() awaitable functions though, for consistency with .execute() and to give me flexibility to change how they work in the future.

I'll also add a block=True option to both of them which causes the function to wait for the write to be successfully executed - defaults to False (fire-and-forget mode).