html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/pull/683#issuecomment-590681676,https://api.github.com/repos/simonw/datasette/issues/683,590681676,MDEyOklzc3VlQ29tbWVudDU5MDY4MTY3Ng==,9599,simonw,2020-02-25T04:48:29Z,2020-02-25T04:48:29Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590679273,https://api.github.com/repos/simonw/datasette/issues/683,590679273,MDEyOklzc3VlQ29tbWVudDU5MDY3OTI3Mw==,9599,simonw,2020-02-25T04:37:21Z,2020-02-25T04:37:21Z,OWNER,I'm happy with this now. I'm going to merge to master.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590617822,https://api.github.com/repos/simonw/datasette/issues/683,590617822,MDEyOklzc3VlQ29tbWVudDU5MDYxNzgyMg==,9599,simonw,2020-02-25T00:26:48Z,2020-02-25T00:26:48Z,OWNER,"This failing test is a nasty one - the whole thing just hangs (so I imagine Travis will run for a while before hopefully giving up). Here's what happens if I add `--full-trace` and then hit Ctrl+C to cancel a test run:
```
$ pytest -k test_execute_write_fn_block_true --full-trace
=================================================================== test session starts ===================================================================
platform darwin -- Python 3.7.5, pytest-5.2.4, py-1.8.1, pluggy-0.13.1
rootdir: /Users/simonw/Dropbox/Development/datasette, inifile: pytest.ini
plugins: asyncio-0.10.0
collected 410 items / 409 deselected / 1 selected
tests/test_database.py ^C^C
================================================================= 409 deselected in 4.45s =================================================================
Traceback (most recent call last):
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py"", line 193, in wrap_session
session.exitstatus = doit(config, session) or 0
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py"", line 237, in _main
config.hook.pytest_runtestloop(session=session)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py"", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 87, in
firstresult=hook.spec.opts.get(""firstresult"") if hook.spec else False,
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 208, in _multicall
return outcome.get_result()
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 80, in get_result
raise ex[1].with_traceback(ex[2])
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 187, in _multicall
res = hook_impl.function(*args)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py"", line 258, in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py"", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 87, in
firstresult=hook.spec.opts.get(""firstresult"") if hook.spec else False,
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 208, in _multicall
return outcome.get_result()
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 80, in get_result
raise ex[1].with_traceback(ex[2])
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 187, in _multicall
res = hook_impl.function(*args)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py"", line 80, in pytest_runtest_protocol
runtestprotocol(item, nextitem=nextitem)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py"", line 95, in runtestprotocol
reports.append(call_and_report(item, ""call"", log))
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py"", line 176, in call_and_report
call = call_runtest_hook(item, when, **kwds)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py"", line 201, in call_runtest_hook
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py"", line 229, in from_call
result = func()
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py"", line 201, in
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py"", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 87, in
firstresult=hook.spec.opts.get(""firstresult"") if hook.spec else False,
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 208, in _multicall
return outcome.get_result()
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 80, in get_result
raise ex[1].with_traceback(ex[2])
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 187, in _multicall
res = hook_impl.function(*args)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py"", line 125, in pytest_runtest_call
item.runtest()
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/python.py"", line 1429, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py"", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 87, in
firstresult=hook.spec.opts.get(""firstresult"") if hook.spec else False,
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 208, in _multicall
return outcome.get_result()
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 80, in get_result
raise ex[1].with_traceback(ex[2])
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 187, in _multicall
res = hook_impl.function(*args)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pytest_asyncio/plugin.py"", line 158, in pytest_pyfunc_call
pyfuncitem.obj(**testargs), loop=event_loop))
File ""/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py"", line 566, in run_until_complete
self.run_forever()
File ""/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py"", line 534, in run_forever
self._run_once()
File ""/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py"", line 1735, in _run_once
event_list = self._selector.select(timeout)
File ""/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/selectors.py"", line 558, in select
kev_list = self._selector.control(None, max_ev, timeout)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/bin/pytest"", line 8, in
sys.exit(main())
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/config/__init__.py"", line 90, in main
return config.hook.pytest_cmdline_main(config=config)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py"", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 87, in
firstresult=hook.spec.opts.get(""firstresult"") if hook.spec else False,
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 208, in _multicall
return outcome.get_result()
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 80, in get_result
raise ex[1].with_traceback(ex[2])
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 187, in _multicall
res = hook_impl.function(*args)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py"", line 230, in pytest_cmdline_main
return wrap_session(config, _main)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py"", line 209, in wrap_session
config.hook.pytest_keyboard_interrupt(excinfo=excinfo)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py"", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 87, in
firstresult=hook.spec.opts.get(""firstresult"") if hook.spec else False,
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 208, in _multicall
return outcome.get_result()
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 80, in get_result
raise ex[1].with_traceback(ex[2])
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py"", line 187, in _multicall
res = hook_impl.function(*args)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/terminal.py"", line 680, in pytest_keyboard_interrupt
self._keyboardinterrupt_memo = excinfo.getrepr(funcargs=True)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py"", line 598, in getrepr
return fmt.repr_excinfo(self)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py"", line 830, in repr_excinfo
reprtraceback = self.repr_traceback(excinfo)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py"", line 778, in repr_traceback
reprentry = self.repr_traceback_entry(entry, einfo)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py"", line 737, in repr_traceback_entry
reprargs = self.repr_args(entry) if not short else None
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py"", line 656, in repr_args
args.append((argname, saferepr(argvalue)))
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py"", line 67, in saferepr
return SafeRepr(maxsize).repr(obj)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py"", line 36, in repr
s = super().repr(x)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py"", line 52, in repr
return self.repr1(x, self.maxlevel)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py"", line 60, in repr1
return getattr(self, 'repr_' + typename)(x, level)
File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py"", line 112, in repr_dict
for key in islice(_possibly_sorted(x), self.maxdict):
KeyboardInterrupt
Task was destroyed but it is pending!
task: wait_for=()]>>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590614896,https://api.github.com/repos/simonw/datasette/issues/683,590614896,MDEyOklzc3VlQ29tbWVudDU5MDYxNDg5Ng==,9599,simonw,2020-02-25T00:16:51Z,2020-02-25T00:16:51Z,OWNER,"The other problem with the poll-for-UUID-completion idea: how long does this mean Datasette needs to keep holding onto the `WriteTask` objects?
Maybe we say you only get to ask ""is this UUID still in the queue"" and if the answer is ""no"" then you assume the task has been completed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590610180,https://api.github.com/repos/simonw/datasette/issues/683,590610180,MDEyOklzc3VlQ29tbWVudDU5MDYxMDE4MA==,9599,simonw,2020-02-25T00:00:07Z,2020-02-25T00:00:07Z,OWNER,"Basic stuff to cover in unit tests:
- Exercise `.execute_write(sql)` - both with block=True and block=False
- Exercise `.execute_write_fn(fn)` in the same way
- Throw 10 updates in the queue, block on just the last one, check it worked correctly
I'm going to write these tests directly against a `Database()` object rather than booting up an entire Datasette instance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590608228,https://api.github.com/repos/simonw/datasette/issues/683,590608228,MDEyOklzc3VlQ29tbWVudDU5MDYwODIyOA==,9599,simonw,2020-02-24T23:52:35Z,2020-02-24T23:52:35Z,OWNER,I'm going to punt on the ability to introspect the write queue and poll for completion using a UUID for the moment. Can add those later.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590607385,https://api.github.com/repos/simonw/datasette/issues/683,590607385,MDEyOklzc3VlQ29tbWVudDU5MDYwNzM4NQ==,9599,simonw,2020-02-24T23:49:37Z,2020-02-24T23:49:37Z,OWNER,"Here's the `upload_csv.py` plugin file I've been playing with:
```python
from datasette import hookimpl
from starlette.responses import PlainTextResponse, HTMLResponse
from starlette.endpoints import HTTPEndpoint
import csv as csv_std
import codecs
import sqlite_utils
class UploadApp(HTTPEndpoint):
def __init__(self, scope, receive, send, datasette):
self.datasette = datasette
super().__init__(scope, receive, send)
def get_database(self):
# For the moment just use the first one that's not immutable
mutable = [db for db in self.datasette.databases.values() if db.is_mutable]
return mutable[0]
async def get(self, request):
return HTMLResponse(
await self.datasette.render_template(
""upload_csv.html"", {""database_name"": self.get_database().name}
)
)
async def post(self, request):
formdata = await request.form()
csv = formdata[""csv""]
# csv.file is a SpooledTemporaryFile, I can read it directly
filename = csv.filename
# TODO: Support other encodings:
reader = csv_std.reader(codecs.iterdecode(csv.file, ""utf-8""))
headers = next(reader)
docs = (dict(zip(headers, row)) for row in reader)
if filename.endswith("".csv""):
filename = filename[:-4]
# Import data into a table of that name using sqlite-utils
db = self.get_database()
def fn(conn):
writable_conn = sqlite_utils.Database(db.path)
writable_conn[filename].insert_all(docs, alter=True)
return writable_conn[filename].count
# Without block=True we may attempt 'select count(*) from ...'
# before the table has been created by the write thread
count = await db.execute_write_fn(fn, block=True)
return HTMLResponse(
await self.datasette.render_template(
""upload_csv_done.html"",
{
""database"": self.get_database().name,
""table"": filename,
""num_docs"": count,
},
)
)
@hookimpl
def asgi_wrapper(datasette):
def wrap_with_asgi_auth(app):
async def wrapped_app(scope, recieve, send):
if scope[""path""] == ""/-/upload-csv"":
await UploadApp(scope, recieve, send, datasette)
else:
await app(scope, recieve, send)
return wrapped_app
return wrap_with_asgi_auth
```
I also dropped copies of the two template files from https://github.com/simonw/datasette-upload-csvs/tree/699e6ca591f36264bfc8e590d877e6852f274beb/datasette_upload_csvs/templates into my `write-templates/` directory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590606825,https://api.github.com/repos/simonw/datasette/issues/683,590606825,MDEyOklzc3VlQ29tbWVudDU5MDYwNjgyNQ==,9599,simonw,2020-02-24T23:47:38Z,2020-02-24T23:47:38Z,OWNER,"Another demo plugin: `delete_table.py`
```python
from datasette import hookimpl
from datasette.utils import escape_sqlite
from starlette.responses import HTMLResponse
from starlette.endpoints import HTTPEndpoint
class DeleteTableApp(HTTPEndpoint):
def __init__(self, scope, receive, send, datasette):
self.datasette = datasette
super().__init__(scope, receive, send)
async def post(self, request):
formdata = await request.form()
database = formdata[""database""]
db = self.datasette.databases[database]
await db.execute_write(""drop table {}"".format(escape_sqlite(formdata[""table""])))
return HTMLResponse(""Table has been deleted."")
@hookimpl
def asgi_wrapper(datasette):
def wrap_with_asgi_auth(app):
async def wrapped_app(scope, recieve, send):
if scope[""path""] == ""/-/delete-table"":
await DeleteTableApp(scope, recieve, send, datasette)
else:
await app(scope, recieve, send)
return wrapped_app
return wrap_with_asgi_auth
```
Then I saved this as `table.html` in the `write-templates/` directory:
```html+django
{% extends ""default:table.html"" %}
{% block content %}
{{ super() }}
{% endblock %}
```
(Needs CSRF protection added)
I ran Datasette like this:
$ datasette --plugins-dir=write-plugins/ data.db --template-dir=write-templates/
Result: I can delete tables!
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590599257,https://api.github.com/repos/simonw/datasette/issues/683,590599257,MDEyOklzc3VlQ29tbWVudDU5MDU5OTI1Nw==,9599,simonw,2020-02-24T23:21:56Z,2020-02-24T23:22:35Z,OWNER,"Also: are UUIDs really necessary here or could I use a simpler form of task identifier? Like an in-memory counter variable that starts at 0 and increments every time this instance of Datasette issues a new task ID?
The neat thing about UUIDs is that I don't have to worry if there are multiple Datasette instances accepting writes behind a load balancer. That seems pretty unlikely (especially considering SQLite databases encourage only one process to be writing at a time)... but I am experimenting with PostgreSQL support in #670 so it's probably worth ensuring these task IDs really are globally unique.
I'm going to stick with UUIDs. They're short-lived enough that their size doesn't really matter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590598689,https://api.github.com/repos/simonw/datasette/issues/683,590598689,MDEyOklzc3VlQ29tbWVudDU5MDU5ODY4OQ==,9599,simonw,2020-02-24T23:20:11Z,2020-02-24T23:20:11Z,OWNER,"I think `if block` it makes sense to return the return value of the function that was executed. Without it all I really need to do is return the `uuid` so something could theoretically poll for completion later on.
But is it weird having a function that returns different types depending on if you passed `block=True` or not? Should they be differently named functions?
I'm OK with the `block=True` pattern changing the return value I think.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590598248,https://api.github.com/repos/simonw/datasette/issues/683,590598248,MDEyOklzc3VlQ29tbWVudDU5MDU5ODI0OA==,9599,simonw,2020-02-24T23:18:50Z,2020-02-24T23:18:50Z,OWNER,"I'm not convinced by the return value of the `.execute_write_fn()` method:
https://github.com/simonw/datasette/blob/ab2348280206bde1390b931ae89d372c2f74b87e/datasette/database.py#L79-L83
Do I really need that `WriteResponse` class or can I do something nicer?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590593120,https://api.github.com/repos/simonw/datasette/issues/683,590593120,MDEyOklzc3VlQ29tbWVudDU5MDU5MzEyMA==,9599,simonw,2020-02-24T23:02:30Z,2020-02-24T23:02:30Z,OWNER,"I'm going to muck around with a couple more demo plugins - in particular one derived from [datasette-upload-csvs](https://github.com/simonw/datasette-upload-csvs) - to make sure I'm comfortable with this API - then add a couple of tests and merge it with documentation that warns ""this is still an experimental feature and may change"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590592581,https://api.github.com/repos/simonw/datasette/issues/683,590592581,MDEyOklzc3VlQ29tbWVudDU5MDU5MjU4MQ==,9599,simonw,2020-02-24T23:00:44Z,2020-02-24T23:01:09Z,OWNER,"I've been testing this out by running one-off demo plugins. I saved the following in a file called `write-plugins/log_asgi.py` (it's a hacked around copy of [asgi-log-to-sqlite](https://github.com/simonw/asgi-log-to-sqlite)) and then running `datasette data.db --plugins-dir=write-plugins/`:
```python
from datasette import hookimpl
import sqlite_utils
import time
class AsgiLogToSqliteViaWriteQueue:
lookup_columns = (
""path"",
""user_agent"",
""referer"",
""accept_language"",
""content_type"",
""query_string"",
)
def __init__(self, app, db):
self.app = app
self.db = db
self._tables_ensured = False
async def ensure_tables(self):
def _ensure_tables(conn):
db = sqlite_utils.Database(conn)
for column in self.lookup_columns:
table = ""{}s"".format(column)
if not db[table].exists():
db[table].create({""id"": int, ""name"": str}, pk=""id"")
if ""requests"" not in db.table_names():
db[""requests""].create(
{
""start"": float,
""method"": str,
""path"": int,
""query_string"": int,
""user_agent"": int,
""referer"": int,
""accept_language"": int,
""http_status"": int,
""content_type"": int,
""client_ip"": str,
""duration"": float,
""body_size"": int,
},
foreign_keys=self.lookup_columns,
)
await self.db.execute_write_fn(_ensure_tables)
async def __call__(self, scope, receive, send):
if not self._tables_ensured:
self._tables_ensured = True
await self.ensure_tables()
response_headers = []
body_size = 0
http_status = None
async def wrapped_send(message):
nonlocal body_size, response_headers, http_status
if message[""type""] == ""http.response.start"":
response_headers = message[""headers""]
http_status = message[""status""]
if message[""type""] == ""http.response.body"":
body_size += len(message[""body""])
await send(message)
start = time.time()
await self.app(scope, receive, wrapped_send)
end = time.time()
path = str(scope[""path""])
query_string = None
if scope.get(""query_string""):
query_string = ""?{}"".format(scope[""query_string""].decode(""utf8""))
request_headers = dict(scope.get(""headers"") or [])
referer = header(request_headers, ""referer"")
user_agent = header(request_headers, ""user-agent"")
accept_language = header(request_headers, ""accept-language"")
content_type = header(dict(response_headers), ""content-type"")
def _log_to_database(conn):
db = sqlite_utils.Database(conn)
db[""requests""].insert(
{
""start"": start,
""method"": scope[""method""],
""path"": lookup(db, ""paths"", path),
""query_string"": lookup(db, ""query_strings"", query_string),
""user_agent"": lookup(db, ""user_agents"", user_agent),
""referer"": lookup(db, ""referers"", referer),
""accept_language"": lookup(db, ""accept_languages"", accept_language),
""http_status"": http_status,
""content_type"": lookup(db, ""content_types"", content_type),
""client_ip"": scope.get(""client"", (None, None))[0],
""duration"": end - start,
""body_size"": body_size,
},
alter=True,
foreign_keys=self.lookup_columns,
)
await self.db.execute_write_fn(_log_to_database)
def header(d, name):
return d.get(name.encode(""utf8""), b"""").decode(""utf8"") or None
def lookup(db, table, value):
return db[table].lookup({""name"": value}) if value else None
@hookimpl
def asgi_wrapper(datasette):
def wrap_with_class(app):
return AsgiLogToSqliteViaWriteQueue(
app, next(iter(datasette.databases.values()))
)
return wrap_with_class
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,
https://github.com/simonw/datasette/pull/683#issuecomment-590518182,https://api.github.com/repos/simonw/datasette/issues/683,590518182,MDEyOklzc3VlQ29tbWVudDU5MDUxODE4Mg==,9599,simonw,2020-02-24T19:53:12Z,2020-02-24T19:53:12Z,OWNER,"Next steps are from comment https://github.com/simonw/datasette/issues/682#issuecomment-590517338
> I'm going to move ahead without needing that ability though. I figure SQLite writes are _fast_, and plugins can be trusted to implement just fast writes. So I'm going to support either fire-and-forget writes (they get added to the queue and a task ID is returned) or have the option to block awaiting the completion of the write (using Janus) but let callers decide which version they want. I may add optional timeouts some time in the future.
>
> I am going to make both `execute_write()` and `execute_write_fn()` awaitable functions though, for consistency with `.execute()` and to give me flexibility to change how they work in the future.
>
> I'll also add a `block=True` option to both of them which causes the function to wait for the write to be successfully executed - defaults to `False` (fire-and-forget mode).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428,.execute_write() and .execute_write_fn() methods on Database,