{"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590518182", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590518182, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDUxODE4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T19:53:12Z", "updated_at": "2020-02-24T19:53:12Z", "author_association": "OWNER", "body": "Next steps are from comment https://github.com/simonw/datasette/issues/682#issuecomment-590517338\r\n> I'm going to move ahead without needing that ability though. I figure SQLite writes are _fast_, and plugins can be trusted to implement just fast writes. So I'm going to support either fire-and-forget writes (they get added to the queue and a task ID is returned) or have the option to block awaiting the completion of the write (using Janus) but let callers decide which version they want. I may add optional timeouts some time in the future.\r\n> \r\n> I am going to make both `execute_write()` and `execute_write_fn()` awaitable functions though, for consistency with `.execute()` and to give me flexibility to change how they work in the future.\r\n> \r\n> I'll also add a `block=True` option to both of them which causes the function to wait for the write to be successfully executed - defaults to `False` (fire-and-forget mode).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590592581", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590592581, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDU5MjU4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:00:44Z", "updated_at": "2020-02-24T23:01:09Z", "author_association": "OWNER", "body": "I've been testing this out by running one-off demo plugins. I saved the following in a file called `write-plugins/log_asgi.py` (it's a hacked around copy of [asgi-log-to-sqlite](https://github.com/simonw/asgi-log-to-sqlite)) and then running `datasette data.db --plugins-dir=write-plugins/`:\r\n```python\r\nfrom datasette import hookimpl\r\nimport sqlite_utils\r\nimport time\r\n\r\n\r\nclass AsgiLogToSqliteViaWriteQueue:\r\n lookup_columns = (\r\n \"path\",\r\n \"user_agent\",\r\n \"referer\",\r\n \"accept_language\",\r\n \"content_type\",\r\n \"query_string\",\r\n )\r\n\r\n def __init__(self, app, db):\r\n self.app = app\r\n self.db = db\r\n self._tables_ensured = False\r\n\r\n async def ensure_tables(self):\r\n def _ensure_tables(conn):\r\n db = sqlite_utils.Database(conn)\r\n for column in self.lookup_columns:\r\n table = \"{}s\".format(column)\r\n if not db[table].exists():\r\n db[table].create({\"id\": int, \"name\": str}, pk=\"id\")\r\n if \"requests\" not in db.table_names():\r\n db[\"requests\"].create(\r\n {\r\n \"start\": float,\r\n \"method\": str,\r\n \"path\": int,\r\n \"query_string\": int,\r\n \"user_agent\": int,\r\n \"referer\": int,\r\n \"accept_language\": int,\r\n \"http_status\": int,\r\n \"content_type\": int,\r\n \"client_ip\": str,\r\n \"duration\": float,\r\n \"body_size\": int,\r\n },\r\n foreign_keys=self.lookup_columns,\r\n )\r\n await self.db.execute_write_fn(_ensure_tables)\r\n\r\n async def __call__(self, scope, receive, send):\r\n if not self._tables_ensured:\r\n self._tables_ensured = True\r\n await self.ensure_tables()\r\n\r\n response_headers = []\r\n body_size = 0\r\n http_status = None\r\n\r\n async def wrapped_send(message):\r\n nonlocal body_size, response_headers, http_status\r\n if message[\"type\"] == \"http.response.start\":\r\n response_headers = message[\"headers\"]\r\n http_status = message[\"status\"]\r\n\r\n if message[\"type\"] == \"http.response.body\":\r\n body_size += len(message[\"body\"])\r\n\r\n await send(message)\r\n\r\n start = time.time()\r\n await self.app(scope, receive, wrapped_send)\r\n end = time.time()\r\n\r\n path = str(scope[\"path\"])\r\n query_string = None\r\n if scope.get(\"query_string\"):\r\n query_string = \"?{}\".format(scope[\"query_string\"].decode(\"utf8\"))\r\n\r\n request_headers = dict(scope.get(\"headers\") or [])\r\n\r\n referer = header(request_headers, \"referer\")\r\n user_agent = header(request_headers, \"user-agent\")\r\n accept_language = header(request_headers, \"accept-language\")\r\n\r\n content_type = header(dict(response_headers), \"content-type\")\r\n\r\n def _log_to_database(conn):\r\n db = sqlite_utils.Database(conn)\r\n db[\"requests\"].insert(\r\n {\r\n \"start\": start,\r\n \"method\": scope[\"method\"],\r\n \"path\": lookup(db, \"paths\", path),\r\n \"query_string\": lookup(db, \"query_strings\", query_string),\r\n \"user_agent\": lookup(db, \"user_agents\", user_agent),\r\n \"referer\": lookup(db, \"referers\", referer),\r\n \"accept_language\": lookup(db, \"accept_languages\", accept_language),\r\n \"http_status\": http_status,\r\n \"content_type\": lookup(db, \"content_types\", content_type),\r\n \"client_ip\": scope.get(\"client\", (None, None))[0],\r\n \"duration\": end - start,\r\n \"body_size\": body_size,\r\n },\r\n alter=True,\r\n foreign_keys=self.lookup_columns,\r\n )\r\n\r\n await self.db.execute_write_fn(_log_to_database)\r\n\r\n\r\ndef header(d, name):\r\n return d.get(name.encode(\"utf8\"), b\"\").decode(\"utf8\") or None\r\n\r\n\r\ndef lookup(db, table, value):\r\n return db[table].lookup({\"name\": value}) if value else None\r\n\r\n\r\n@hookimpl\r\ndef asgi_wrapper(datasette):\r\n def wrap_with_class(app):\r\n return AsgiLogToSqliteViaWriteQueue(\r\n app, next(iter(datasette.databases.values()))\r\n )\r\n\r\n return wrap_with_class\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590593120", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590593120, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDU5MzEyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:02:30Z", "updated_at": "2020-02-24T23:02:30Z", "author_association": "OWNER", "body": "I'm going to muck around with a couple more demo plugins - in particular one derived from [datasette-upload-csvs](https://github.com/simonw/datasette-upload-csvs) - to make sure I'm comfortable with this API - then add a couple of tests and merge it with documentation that warns \"this is still an experimental feature and may change\".", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590598248", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590598248, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDU5ODI0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:18:50Z", "updated_at": "2020-02-24T23:18:50Z", "author_association": "OWNER", "body": "I'm not convinced by the return value of the `.execute_write_fn()` method:\r\n\r\nhttps://github.com/simonw/datasette/blob/ab2348280206bde1390b931ae89d372c2f74b87e/datasette/database.py#L79-L83\r\n\r\nDo I really need that `WriteResponse` class or can I do something nicer?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590598689", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590598689, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDU5ODY4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:20:11Z", "updated_at": "2020-02-24T23:20:11Z", "author_association": "OWNER", "body": "I think `if block` it makes sense to return the return value of the function that was executed. Without it all I really need to do is return the `uuid` so something could theoretically poll for completion later on.\r\n\r\nBut is it weird having a function that returns different types depending on if you passed `block=True` or not? Should they be differently named functions?\r\n\r\nI'm OK with the `block=True` pattern changing the return value I think.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590599257", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590599257, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDU5OTI1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:21:56Z", "updated_at": "2020-02-24T23:22:35Z", "author_association": "OWNER", "body": "Also: are UUIDs really necessary here or could I use a simpler form of task identifier? Like an in-memory counter variable that starts at 0 and increments every time this instance of Datasette issues a new task ID?\r\n\r\nThe neat thing about UUIDs is that I don't have to worry if there are multiple Datasette instances accepting writes behind a load balancer. That seems pretty unlikely (especially considering SQLite databases encourage only one process to be writing at a time)... but I am experimenting with PostgreSQL support in #670 so it's probably worth ensuring these task IDs really are globally unique.\r\n\r\nI'm going to stick with UUIDs. They're short-lived enough that their size doesn't really matter.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 570101428, "label": ".execute_write() and .execute_write_fn() methods on Database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/683#issuecomment-590606825", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/683", "id": 590606825, "node_id": "MDEyOklzc3VlQ29tbWVudDU5MDYwNjgyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-02-24T23:47:38Z", "updated_at": "2020-02-24T23:47:38Z", "author_association": "OWNER", "body": "Another demo plugin: `delete_table.py`\r\n```python\r\nfrom datasette import hookimpl\r\nfrom datasette.utils import escape_sqlite\r\nfrom starlette.responses import HTMLResponse\r\nfrom starlette.endpoints import HTTPEndpoint\r\n\r\n\r\nclass DeleteTableApp(HTTPEndpoint):\r\n def __init__(self, scope, receive, send, datasette):\r\n self.datasette = datasette\r\n super().__init__(scope, receive, send)\r\n\r\n async def post(self, request):\r\n formdata = await request.form()\r\n database = formdata[\"database\"]\r\n db = self.datasette.databases[database]\r\n await db.execute_write(\"drop table {}\".format(escape_sqlite(formdata[\"table\"])))\r\n return HTMLResponse(\"Table has been deleted.\")\r\n\r\n\r\n@hookimpl\r\ndef asgi_wrapper(datasette):\r\n def wrap_with_asgi_auth(app):\r\n async def wrapped_app(scope, recieve, send):\r\n if scope[\"path\"] == \"/-/delete-table\":\r\n await DeleteTableApp(scope, recieve, send, datasette)\r\n else:\r\n await app(scope, recieve, send)\r\n\r\n return wrapped_app\r\n\r\n return wrap_with_asgi_auth\r\n```\r\nThen I saved this as `table.html` in the `write-templates/` directory:\r\n```html+django\r\n{% extends \"default:table.html\" %}\r\n\r\n{% block content %}\r\n
\r\n{{ super() }}\r\n{% endblock %}\r\n```\r\n(Needs CSRF protection added)\r\n\r\nI ran Datasette like this:\r\n\r\n $ datasette --plugins-dir=write-plugins/ data.db --template-dir=write-templates/\r\n\r\nResult: I can delete tables!\r\n\r\n