issue_comments

14 rows where issue = 570101428 sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date), updated_at (date)

user

issue

  • .execute_write() and .execute_write_fn() methods on Database · 14

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
590681676 https://github.com/simonw/datasette/pull/683#issuecomment-590681676 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDY4MTY3Ng== simonw 9599 2020-02-25T04:48:29Z 2020-02-25T04:48:29Z OWNER

Documentation: https://datasette.readthedocs.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590679273 https://github.com/simonw/datasette/pull/683#issuecomment-590679273 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDY3OTI3Mw== simonw 9599 2020-02-25T04:37:21Z 2020-02-25T04:37:21Z OWNER

I'm happy with this now. I'm going to merge to master.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590617822 https://github.com/simonw/datasette/pull/683#issuecomment-590617822 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDYxNzgyMg== simonw 9599 2020-02-25T00:26:48Z 2020-02-25T00:26:48Z OWNER

This failing test is a nasty one - the whole thing just hangs (so I imagine Travis will run for a while before hopefully giving up). Here's what happens if I add --full-trace and then hit Ctrl+C to cancel a test run:

$ pytest -k test_execute_write_fn_block_true --full-trace
=================================================================== test session starts ===================================================================
platform darwin -- Python 3.7.5, pytest-5.2.4, py-1.8.1, pluggy-0.13.1
rootdir: /Users/simonw/Dropbox/Development/datasette, inifile: pytest.ini
plugins: asyncio-0.10.0
collected 410 items / 409 deselected / 1 selected                                                                                                         

tests/test_database.py ^C^C

================================================================= 409 deselected in 4.45s =================================================================
Traceback (most recent call last):
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 193, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 237, in _main
    config.hook.pytest_runtestloop(session=session)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 258, in pytest_runtestloop
    item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 80, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 95, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 176, in call_and_report
    call = call_runtest_hook(item, when, **kwds)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 201, in call_runtest_hook
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 229, in from_call
    result = func()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 201, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 125, in pytest_runtest_call
    item.runtest()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/python.py", line 1429, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pytest_asyncio/plugin.py", line 158, in pytest_pyfunc_call
    pyfuncitem.obj(**testargs), loop=event_loop))
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 566, in run_until_complete
    self.run_forever()
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 534, in run_forever
    self._run_once()
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 1735, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/selectors.py", line 558, in select
    kev_list = self._selector.control(None, max_ev, timeout)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/bin/pytest", line 8, in <module>
    sys.exit(main())
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/config/__init__.py", line 90, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 230, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 209, in wrap_session
    config.hook.pytest_keyboard_interrupt(excinfo=excinfo)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/terminal.py", line 680, in pytest_keyboard_interrupt
    self._keyboardinterrupt_memo = excinfo.getrepr(funcargs=True)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 598, in getrepr
    return fmt.repr_excinfo(self)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 830, in repr_excinfo
    reprtraceback = self.repr_traceback(excinfo)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 778, in repr_traceback
    reprentry = self.repr_traceback_entry(entry, einfo)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 737, in repr_traceback_entry
    reprargs = self.repr_args(entry) if not short else None
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 656, in repr_args
    args.append((argname, saferepr(argvalue)))
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py", line 67, in saferepr
    return SafeRepr(maxsize).repr(obj)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py", line 36, in repr
    s = super().repr(x)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 52, in repr
    return self.repr1(x, self.maxlevel)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 60, in repr1
    return getattr(self, 'repr_' + typename)(x, level)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 112, in repr_dict
    for key in islice(_possibly_sorted(x), self.maxdict):
KeyboardInterrupt
Task was destroyed but it is pending!
task: <Task pending coro=<test_execute_write_fn_block_true() running at /Users/simonw/Dropbox/Development/datasette/tests/test_database.py:139> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x105580510>()]>>
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590614896 https://github.com/simonw/datasette/pull/683#issuecomment-590614896 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDYxNDg5Ng== simonw 9599 2020-02-25T00:16:51Z 2020-02-25T00:16:51Z OWNER

The other problem with the poll-for-UUID-completion idea: how long does this mean Datasette needs to keep holding onto the WriteTask objects?

Maybe we say you only get to ask "is this UUID still in the queue" and if the answer is "no" then you assume the task has been completed.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590610180 https://github.com/simonw/datasette/pull/683#issuecomment-590610180 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDYxMDE4MA== simonw 9599 2020-02-25T00:00:07Z 2020-02-25T00:00:07Z OWNER

Basic stuff to cover in unit tests:
- Exercise .execute_write(sql) - both with block=True and block=False
- Exercise .execute_write_fn(fn) in the same way
- Throw 10 updates in the queue, block on just the last one, check it worked correctly

I'm going to write these tests directly against a Database() object rather than booting up an entire Datasette instance.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590608228 https://github.com/simonw/datasette/pull/683#issuecomment-590608228 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDYwODIyOA== simonw 9599 2020-02-24T23:52:35Z 2020-02-24T23:52:35Z OWNER

I'm going to punt on the ability to introspect the write queue and poll for completion using a UUID for the moment. Can add those later.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590607385 https://github.com/simonw/datasette/pull/683#issuecomment-590607385 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDYwNzM4NQ== simonw 9599 2020-02-24T23:49:37Z 2020-02-24T23:49:37Z OWNER

Here's the upload_csv.py plugin file I've been playing with:

from datasette import hookimpl
from starlette.responses import PlainTextResponse, HTMLResponse
from starlette.endpoints import HTTPEndpoint
import csv as csv_std
import codecs
import sqlite_utils


class UploadApp(HTTPEndpoint):
    def __init__(self, scope, receive, send, datasette):
        self.datasette = datasette
        super().__init__(scope, receive, send)

    def get_database(self):
        # For the moment just use the first one that's not immutable
        mutable = [db for db in self.datasette.databases.values() if db.is_mutable]
        return mutable[0]

    async def get(self, request):
        return HTMLResponse(
            await self.datasette.render_template(
                "upload_csv.html", {"database_name": self.get_database().name}
            )
        )

    async def post(self, request):
        formdata = await request.form()
        csv = formdata["csv"]
        # csv.file is a SpooledTemporaryFile, I can read it directly
        filename = csv.filename
        # TODO: Support other encodings:
        reader = csv_std.reader(codecs.iterdecode(csv.file, "utf-8"))
        headers = next(reader)
        docs = (dict(zip(headers, row)) for row in reader)
        if filename.endswith(".csv"):
            filename = filename[:-4]
        # Import data into a table of that name using sqlite-utils
        db = self.get_database()

        def fn(conn):
            writable_conn = sqlite_utils.Database(db.path)
            writable_conn[filename].insert_all(docs, alter=True)
            return writable_conn[filename].count

        # Without block=True we may attempt 'select count(*) from ...'
        # before the table has been created by the write thread
        count = await db.execute_write_fn(fn, block=True)

        return HTMLResponse(
            await self.datasette.render_template(
                "upload_csv_done.html",
                {
                    "database": self.get_database().name,
                    "table": filename,
                    "num_docs": count,
                },
            )
        )


@hookimpl
def asgi_wrapper(datasette):
    def wrap_with_asgi_auth(app):
        async def wrapped_app(scope, recieve, send):
            if scope["path"] == "/-/upload-csv":
                await UploadApp(scope, recieve, send, datasette)
            else:
                await app(scope, recieve, send)

        return wrapped_app

    return wrap_with_asgi_auth

I also dropped copies of the two template files from https://github.com/simonw/datasette-upload-csvs/tree/699e6ca591f36264bfc8e590d877e6852f274beb/datasette_upload_csvs/templates into my write-templates/ directory.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590606825 https://github.com/simonw/datasette/pull/683#issuecomment-590606825 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDYwNjgyNQ== simonw 9599 2020-02-24T23:47:38Z 2020-02-24T23:47:38Z OWNER

Another demo plugin: delete_table.py

from datasette import hookimpl
from datasette.utils import escape_sqlite
from starlette.responses import HTMLResponse
from starlette.endpoints import HTTPEndpoint


class DeleteTableApp(HTTPEndpoint):
    def __init__(self, scope, receive, send, datasette):
        self.datasette = datasette
        super().__init__(scope, receive, send)

    async def post(self, request):
        formdata = await request.form()
        database = formdata["database"]
        db = self.datasette.databases[database]
        await db.execute_write("drop table {}".format(escape_sqlite(formdata["table"])))
        return HTMLResponse("Table has been deleted.")


@hookimpl
def asgi_wrapper(datasette):
    def wrap_with_asgi_auth(app):
        async def wrapped_app(scope, recieve, send):
            if scope["path"] == "/-/delete-table":
                await DeleteTableApp(scope, recieve, send, datasette)
            else:
                await app(scope, recieve, send)

        return wrapped_app

    return wrap_with_asgi_auth

Then I saved this as table.html in the write-templates/ directory:

{% extends "default:table.html" %}

{% block content %}
<form action="/-/delete-table" method="POST">
    <p>
        <input type="hidden" name="database" value="{{ database }}">
        <input type="hidden" name="table" value="{{ table }}">
        <input type="submit" value="Delete this table">
    </p>
</form>
{{ super() }}
{% endblock %}

(Needs CSRF protection added)

I ran Datasette like this:

$ datasette --plugins-dir=write-plugins/ data.db --template-dir=write-templates/

Result: I can delete tables!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590599257 https://github.com/simonw/datasette/pull/683#issuecomment-590599257 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDU5OTI1Nw== simonw 9599 2020-02-24T23:21:56Z 2020-02-24T23:22:35Z OWNER

Also: are UUIDs really necessary here or could I use a simpler form of task identifier? Like an in-memory counter variable that starts at 0 and increments every time this instance of Datasette issues a new task ID?

The neat thing about UUIDs is that I don't have to worry if there are multiple Datasette instances accepting writes behind a load balancer. That seems pretty unlikely (especially considering SQLite databases encourage only one process to be writing at a time)... but I am experimenting with PostgreSQL support in #670 so it's probably worth ensuring these task IDs really are globally unique.

I'm going to stick with UUIDs. They're short-lived enough that their size doesn't really matter.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590598689 https://github.com/simonw/datasette/pull/683#issuecomment-590598689 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDU5ODY4OQ== simonw 9599 2020-02-24T23:20:11Z 2020-02-24T23:20:11Z OWNER

I think if block it makes sense to return the return value of the function that was executed. Without it all I really need to do is return the uuid so something could theoretically poll for completion later on.

But is it weird having a function that returns different types depending on if you passed block=True or not? Should they be differently named functions?

I'm OK with the block=True pattern changing the return value I think.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590598248 https://github.com/simonw/datasette/pull/683#issuecomment-590598248 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDU5ODI0OA== simonw 9599 2020-02-24T23:18:50Z 2020-02-24T23:18:50Z OWNER

I'm not convinced by the return value of the .execute_write_fn() method:

https://github.com/simonw/datasette/blob/ab2348280206bde1390b931ae89d372c2f74b87e/datasette/database.py#L79-L83

Do I really need that WriteResponse class or can I do something nicer?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590593120 https://github.com/simonw/datasette/pull/683#issuecomment-590593120 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDU5MzEyMA== simonw 9599 2020-02-24T23:02:30Z 2020-02-24T23:02:30Z OWNER

I'm going to muck around with a couple more demo plugins - in particular one derived from datasette-upload-csvs - to make sure I'm comfortable with this API - then add a couple of tests and merge it with documentation that warns "this is still an experimental feature and may change".

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590592581 https://github.com/simonw/datasette/pull/683#issuecomment-590592581 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDU5MjU4MQ== simonw 9599 2020-02-24T23:00:44Z 2020-02-24T23:01:09Z OWNER

I've been testing this out by running one-off demo plugins. I saved the following in a file called write-plugins/log_asgi.py (it's a hacked around copy of asgi-log-to-sqlite) and then running datasette data.db --plugins-dir=write-plugins/:

from datasette import hookimpl
import sqlite_utils
import time


class AsgiLogToSqliteViaWriteQueue:
    lookup_columns = (
        "path",
        "user_agent",
        "referer",
        "accept_language",
        "content_type",
        "query_string",
    )

    def __init__(self, app, db):
        self.app = app
        self.db = db
        self._tables_ensured = False

    async def ensure_tables(self):
        def _ensure_tables(conn):
            db = sqlite_utils.Database(conn)
            for column in self.lookup_columns:
                table = "{}s".format(column)
                if not db[table].exists():
                    db[table].create({"id": int, "name": str}, pk="id")
            if "requests" not in db.table_names():
                db["requests"].create(
                    {
                        "start": float,
                        "method": str,
                        "path": int,
                        "query_string": int,
                        "user_agent": int,
                        "referer": int,
                        "accept_language": int,
                        "http_status": int,
                        "content_type": int,
                        "client_ip": str,
                        "duration": float,
                        "body_size": int,
                    },
                    foreign_keys=self.lookup_columns,
                )
        await self.db.execute_write_fn(_ensure_tables)

    async def __call__(self, scope, receive, send):
        if not self._tables_ensured:
            self._tables_ensured = True
            await self.ensure_tables()

        response_headers = []
        body_size = 0
        http_status = None

        async def wrapped_send(message):
            nonlocal body_size, response_headers, http_status
            if message["type"] == "http.response.start":
                response_headers = message["headers"]
                http_status = message["status"]

            if message["type"] == "http.response.body":
                body_size += len(message["body"])

            await send(message)

        start = time.time()
        await self.app(scope, receive, wrapped_send)
        end = time.time()

        path = str(scope["path"])
        query_string = None
        if scope.get("query_string"):
            query_string = "?{}".format(scope["query_string"].decode("utf8"))

        request_headers = dict(scope.get("headers") or [])

        referer = header(request_headers, "referer")
        user_agent = header(request_headers, "user-agent")
        accept_language = header(request_headers, "accept-language")

        content_type = header(dict(response_headers), "content-type")

        def _log_to_database(conn):
            db = sqlite_utils.Database(conn)
            db["requests"].insert(
                {
                    "start": start,
                    "method": scope["method"],
                    "path": lookup(db, "paths", path),
                    "query_string": lookup(db, "query_strings", query_string),
                    "user_agent": lookup(db, "user_agents", user_agent),
                    "referer": lookup(db, "referers", referer),
                    "accept_language": lookup(db, "accept_languages", accept_language),
                    "http_status": http_status,
                    "content_type": lookup(db, "content_types", content_type),
                    "client_ip": scope.get("client", (None, None))[0],
                    "duration": end - start,
                    "body_size": body_size,
                },
                alter=True,
                foreign_keys=self.lookup_columns,
            )

        await self.db.execute_write_fn(_log_to_database)


def header(d, name):
    return d.get(name.encode("utf8"), b"").decode("utf8") or None


def lookup(db, table, value):
    return db[table].lookup({"name": value}) if value else None


@hookimpl
def asgi_wrapper(datasette):
    def wrap_with_class(app):
        return AsgiLogToSqliteViaWriteQueue(
            app, next(iter(datasette.databases.values()))
        )

    return wrap_with_class
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428
590518182 https://github.com/simonw/datasette/pull/683#issuecomment-590518182 https://api.github.com/repos/simonw/datasette/issues/683 MDEyOklzc3VlQ29tbWVudDU5MDUxODE4Mg== simonw 9599 2020-02-24T19:53:12Z 2020-02-24T19:53:12Z OWNER

Next steps are from comment https://github.com/simonw/datasette/issues/682#issuecomment-590517338

I'm going to move ahead without needing that ability though. I figure SQLite writes are fast, and plugins can be trusted to implement just fast writes. So I'm going to support either fire-and-forget writes (they get added to the queue and a task ID is returned) or have the option to block awaiting the completion of the write (using Janus) but let callers decide which version they want. I may add optional timeouts some time in the future.

I am going to make both execute_write() and execute_write_fn() awaitable functions though, for consistency with .execute() and to give me flexibility to change how they work in the future.

I'll also add a block=True option to both of them which causes the function to wait for the write to be successfully executed - defaults to False (fire-and-forget mode).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.execute_write() and .execute_write_fn() methods on Database 570101428

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 30.652ms · About: github-to-sqlite