Test ran in 38 seconds and passed! https://github.com/simonw/datasette/runs/6265954274?check_suite_focus=true

I'm going to have it run on every commit and PR.

https://github.com/simonw/datasette/runs/6265915080?check_suite_focus=true failed but looks like it passed because I forgot to use set -e at the start of the bash script.

It failed because it didn't have build available.

I'm going to start off by running this manually - I may run it on every commit once this is all a little bit more stable.

I can base the workflow on https://github.com/simonw/scrape-hacker-news-by-domain/blob/main/.github/workflows/scrape.yml

Here's a script that seems to work. It builds the wheel, starts a Python web server that serves the wheel, runs a test with shot-scraper and then shuts down the server again.

```bash

!/bin/bash

Build the wheel

python3 -m build

Find name of wheel

wheel=$(basename $(ls dist/*.whl))

strip off the dist/

Create a blank index page

echo '

' > dist/index.html

Run a server for that dist/ folder

cd dist python3 -m http.server 8529 & cd ..

shot-scraper javascript http://localhost:8529/ " async () => { let pyodide = await loadPyodide(); await pyodide.loadPackage(['micropip', 'ssl', 'setuptools']); let output = await pyodide.runPythonAsync(` import micropip await micropip.install('h11==0.12.0') await micropip.install('http://localhost:8529/$wheel') import ssl import setuptools from datasette.app import Datasette ds = Datasette(memory=True, settings={'num_sql_threads': 0}) (await ds.client.get('/_memory.json?sql=select+55+as+itworks&_shape=array')).text `); if (JSON.parse(output)[0].itworks != 55) { throw 'Got ' + output + ', expected itworks: 55'; } return 'Test passed!'; } "

Shut down the server

pkill -f 'http.server 8529' ```

That alpha release works!

https://pyodide.org/en/stable/console.html

```pycon Welcome to the Pyodide terminal emulator 🐍 Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM Type "help", "copyright", "credits" or "license" for more information.

import micropip await micropip.install("datasette==0.62a0") import ssl import setuptools from datasette.app import Datasette ds = Datasette(memory=True, settings={"num_sql_threads": 0}) await ds.client.get("/.json") <Response [200 OK]> (await ds.client.get("/.json")).json() {'_memory': {'name': '_memory', 'hash': None, 'color': 'a6c7b9', 'path': '/_memory', 'tables_and_views_truncated': [], 'tab les_and_views_more': False, 'tables_count': 0, 'table_rows_sum': 0, 'show_table_row_counts': False, 'hidden_table_rows_sum' : 0, 'hidden_tables_count': 0, 'views_count': 0, 'private': False}}

```

This is good enough to push an alpha.

I got a build from the pyodide branch to work!

``` Welcome to the Pyodide terminal emulator 🐍 Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM Type "help", "copyright", "credits" or "license" for more information.

import micropip await micropip.install("https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl") Traceback (most recent call last): File "<console>", line 1, in <module> File "/lib/python3.10/asyncio/futures.py", line 284, in await yield self # This tells Task to wait for completion. File "/lib/python3.10/asyncio/tasks.py", line 304, in wakeup future.result() File "/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception File "/lib/python3.10/asyncio/tasks.py", line 234, in __step result = coro.throw(exc) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 183, in install transaction = await self.gather_requirements(requirements, ctx, keep_going) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 173, in gather_requirements await gather(requirement_promises) File "/lib/python3.10/asyncio/futures.py", line 284, in await yield self # This tells Task to wait for completion. File "/lib/python3.10/asyncio/tasks.py", line 304, in wakeup future.result() File "/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception File "/lib/python3.10/asyncio/tasks.py", line 232, in __step result = coro.send(None) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 245, in add_requirement await self.add_wheel(name, wheel, version, (), ctx, transaction) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement await self.add_wheel( File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement await self.add_wheel( File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 276, in add_requirement raise ValueError( ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed await micropip.install("https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl") Traceback (most recent call last): File "<console>", line 1, in <module> File "/lib/python3.10/asyncio/futures.py", line 284, in __await yield self # This tells Task to wait for completion. File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup future.result() File "/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception File "/lib/python3.10/asyncio/tasks.py", line 234, in __step result = coro.throw(exc) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 183, in install transaction = await self.gather_requirements(requirements, ctx, keep_going) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 173, in gather_requirements await gather(requirement_promises) File "/lib/python3.10/asyncio/futures.py", line 284, in __await yield self # This tells Task to wait for completion. File "/lib/python3.10/asyncio/tasks.py", line 304, in wakeup future.result() File "/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception File "/lib/python3.10/asyncio/tasks.py", line 232, in __step result = coro.send(None) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 245, in add_requirement await self.add_wheel(name, wheel, version, (), ctx, transaction) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement await self.add_wheel( File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement await self.add_wheel( File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 276, in add_requirement raise ValueError( ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed await micropip.install("h11==0.12") await micropip.install("https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl") import datasette from datasette.app import Datasette Traceback (most recent call last): File "<console>", line 1, in <module> File "/lib/python3.10/site-packages/datasette/app.py", line 9, in <module> import httpx File "/lib/python3.10/site-packages/httpx/__init.py", line 2, in <module> from ._api import delete, get, head, options, patch, post, put, request, stream File "/lib/python3.10/site-packages/httpx/_api.py", line 4, in <module> from ._client import Client File "/lib/python3.10/site-packages/httpx/_client.py", line 9, in <module> from ._auth import Auth, BasicAuth, FunctionAuth File "/lib/python3.10/site-packages/httpx/_auth.py", line 10, in <module> from ._models import Request, Response File "/lib/python3.10/site-packages/httpx/_models.py", line 16, in <module> from ._content import ByteStream, UnattachedStream, encode_request, encode_response File "/lib/python3.10/site-packages/httpx/_content.py", line 17, in <module> from ._multipart import MultipartStream File "/lib/python3.10/site-packages/httpx/_multipart.py", line 7, in <module> from ._types import ( File "/lib/python3.10/site-packages/httpx/_types.py", line 5, in <module> import ssl File "/lib/python3.10/ssl.py", line 98, in <module> import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl' import ssl from datasette.app import Datasette Traceback (most recent call last): File "<console>", line 1, in <module> File "/lib/python3.10/site-packages/datasette/app.py", line 14, in <module> import pkg_resources ModuleNotFoundError: No module named 'pkg_resources' import setuptools from datasette.app import Datasette ds = Datasette(memory=True) ds <datasette.app.Datasette object at 0x1cc4fb8> await ds.client.get("/") Traceback (most recent call last): File "/lib/python3.10/site-packages/datasette/app.py", line 1268, in route_path response = await view(request, send) File "/lib/python3.10/site-packages/datasette/views/base.py", line 134, in view return await self.dispatch_request(request) File "/lib/python3.10/site-packages/datasette/views/base.py", line 89, in dispatch_request await self.ds.refresh_schemas() File "/lib/python3.10/site-packages/datasette/app.py", line 353, in refresh_schemas await self._refresh_schemas() File "/lib/python3.10/site-packages/datasette/app.py", line 358, in _refresh_schemas await init_internal_db(internal_db) File "/lib/python3.10/site-packages/datasette/utils/internal_db.py", line 65, in init_internal_db await db.execute_write_script(create_tables_sql) File "/lib/python3.10/site-packages/datasette/database.py", line 116, in execute_write_script results = await self.execute_write_fn(_inner, block=block) File "/lib/python3.10/site-packages/datasette/database.py", line 155, in execute_write_fn self._write_thread.start() File "/lib/python3.10/threading.py", line 928, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread <Response [500 Internal Server Error]> ds = Datasette(memory=True, settings={"num_sql_threads": 0}) await ds.client.get("/") <Response [200 OK]> (await ds.client.get("/")).text '<!DOCTYPE html>\n<html>\n<head>\n <title>Datasette: _memory</title>\n <link rel="stylesheet" href="/-/static/app.css ?cead5a">\n <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">\n\n<link rel="alterna te" type="application/json+datasette" href="http://localhost/.json"></head>\n<body class="index">\n

\n<header><nav>\n \n \n</nav></header>\n\n\n\n \n\n\n\n<section class="content">\n\n

Datasette

\n\n\n\n\n\n

r detailsClickedWithin = null;\n while (target && target.tagName != \'DETAILS\') {\n target = target.parentNode;\ n }\n if (target && target.tagName == \'DETAILS\') {\n detailsClickedWithin = target;\n }\n Array.from(d ocument.getElementsByTagName(\'details\')).filter(\n (details) => details.open && details != detailsClickedWithin\n ).forEach(details => details.open = false);\n});\n</script>\n\n\n\n\n</body>\n </html>'

```

That ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed error is annoying. I assume it's a uvicorn dependency clash of some sort, because I wasn't getting that when I removed uvicorn as a dependency.

I can avoid it by running this first though:

await micropip.install("h11==0.12")

This code breaks if that setting is 0:

https://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/app.py#L291-L293

It's used here:

https://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/database.py#L188-L190

I'll release this as a 0.62a0 as soon as it's ready, so I can start testing it out in Pyodide for real.

I'll use my original from 2009: https://www.djangosnippets.org/snippets/1431/

I'm going to vendor it and update the documentation.

I'm going to add a Datasette setting to disable threading entirely, designed for usage in this particular case.

I thought about adding a new setting, then I noticed this:

datasette mydatabase.db --setting num_sql_threads 10

I'm going to let users set that to 0 to disable threaded execution of SQL queries.

Maybe I can leave uvicorn as a dependency? Installing it works OK, it only generates errors when you try to import it:

```pycon Welcome to the Pyodide terminal emulator 🐍 Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM Type "help", "copyright", "credits" or "license" for more information.

import micropip await micropip.install("uvicorn") import uvicorn Traceback (most recent call last): File "<console>", line 1, in <module> File "/lib/python3.10/site-packages/uvicorn/init.py", line 1, in <module> from uvicorn.config import Config File "/lib/python3.10/site-packages/uvicorn/config.py", line 8, in <module> import ssl File "/lib/python3.10/ssl.py", line 98, in <module> import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl' import ssl import uvicorn Traceback (most recent call last): File "<console>", line 1, in <module> File "/lib/python3.10/site-packages/uvicorn/init.py", line 2, in <module> from uvicorn.main import Server, main, run File "/lib/python3.10/site-packages/uvicorn/main.py", line 24, in <module> from uvicorn.supervisors import ChangeReload, Multiprocess File "/lib/python3.10/site-packages/uvicorn/supervisors/init.py", line 3, in <module> from uvicorn.supervisors.basereload import BaseReload File "/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py", line 12, in <module> from uvicorn.subprocess import get_subprocess File "/lib/python3.10/site-packages/uvicorn/subprocess.py", line 14, in <module> multiprocessing.allow_connection_pickling() File "/lib/python3.10/multiprocessing/context.py", line 170, in allow_connection_pickling from . import connection File "/lib/python3.10/multiprocessing/connection.py", line 21, in <module> import _multiprocessing ModuleNotFoundError: No module named '_multiprocessing' import multiprocessing import uvicorn Traceback (most recent call last): File "<console>", line 1, in <module> File "/lib/python3.10/site-packages/uvicorn/init.py", line 2, in <module> from uvicorn.main import Server, main, run File "/lib/python3.10/site-packages/uvicorn/main.py", line 24, in <module> from uvicorn.supervisors import ChangeReload, Multiprocess File "/lib/python3.10/site-packages/uvicorn/supervisors/init.py", line 3, in <module> from uvicorn.supervisors.basereload import BaseReload File "/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py", line 12, in <module> from uvicorn.subprocess import get_subprocess File "/lib/python3.10/site-packages/uvicorn/subprocess.py", line 14, in <module> multiprocessing.allow_connection_pickling() File "/lib/python3.10/multiprocessing/context.py", line 170, in allow_connection_pickling from . import connection File "/lib/python3.10/multiprocessing/connection.py", line 21, in <module> import _multiprocessing ModuleNotFoundError: No module named '_multiprocessing'

`` Since theimport ssltrick fixed the_sslerror I was hopeful thatimport multiprocessingcould fix the_multiprocessing` one, but sadly it did not.

But it looks like i can address this issue just by making import uvicorn in app.py an optional import.

Here's the full diff I applied to Datasette to get it fully working in Pyodide:

https://github.com/simonw/datasette/compare/94a3171b01fde5c52697aeeff052e3ad4bab5391...8af32bc5b03c30b1f7a4a8cc4bd80eb7e2ee7b81

And as a visible diff:

```diff diff --git a/datasette/app.py b/datasette/app.py index d269372..6c0c5fc 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -15,7 +15,6 @@ import pkg_resources import re import secrets import sys -import threading import traceback import urllib.parse from concurrent import futures @@ -26,7 +25,6 @@ from itsdangerous import URLSafeSerializer from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound -import uvicorn

from .views.base import DatasetteError, ureg from .views.database import DatabaseDownload, DatabaseView @@ -813,7 +811,6 @@ class Datasette: }, "datasette": datasette_version, "asgi": "3.0", - "uvicorn": uvicorn.version, "sqlite": { "version": sqlite_version, "fts_versions": fts_versions, @@ -854,23 +851,7 @@ class Datasette: ]

 def _threads(self):

• threads = list(threading.enumerate())
• d = {
• "num_threads": len(threads),
• "threads": [
• {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads
• ],
• }

• Only available in Python 3.7+

• if hasattr(asyncio, "all_tasks"):
• tasks = asyncio.all_tasks()
• d.update(
• {
• "num_tasks": len(tasks),
• "tasks": [_cleaner_task_str(t) for t in tasks],
• }
• )
• return d

• return {"num_threads": 0, "threads": []}

  def _actor(self, request): return {"actor": request.actor} diff --git a/datasette/database.py b/datasette/database.py index ba594a8..b50142d 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -4,7 +4,6 @@ from pathlib import Path import janus import queue import sys -import threading import uuid

from .tracer import trace @@ -21,8 +20,6 @@ from .utils import ( ) from .inspect import inspect_hash

-connections = threading.local()

AttachedDatabase = namedtuple("AttachedDatabase", ("seq", "name", "file"))

@@ -43,12 +40,12 @@ class Database: self.hash = None self.cached_size = None self._cached_table_counts = None - self._write_thread = None - self._write_queue = None if not self.is_mutable and not self.is_memory: p = Path(path) self.hash = inspect_hash(p) self.cached_size = p.stat().st_size + self._read_connection = None + self._write_connection = None

 @property
 def cached_table_counts(self):

@@ -134,60 +131,17 @@ class Database: return results

 async def execute_write_fn(self, fn, block=True):

• task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io")
• if self._write_queue is None:
• self._write_queue = queue.Queue()
• if self._write_thread is None:
• self._write_thread = threading.Thread(
• target=self._execute_writes, daemon=True
• )
• self._write_thread.start()
• reply_queue = janus.Queue()
• self._write_queue.put(WriteTask(fn, task_id, reply_queue))
• if block:
• result = await reply_queue.async_q.get()
• if isinstance(result, Exception):
• raise result
• else:
• return result
• else:

• return task_id

• def _execute_writes(self):

• Infinite looping thread that protects the single write connection

• to this database

• conn_exception = None
• conn = None
• try:
• conn = self.connect(write=True)
• self.ds._prepare_connection(conn, self.name)
• except Exception as e:
• conn_exception = e
• while True:
• task = self._write_queue.get()
• if conn_exception is not None:
• result = conn_exception
• else:
• try:
• result = task.fn(conn)
• except Exception as e:
• sys.stderr.write("{}\n".format(e))
• sys.stderr.flush()
• result = e
• task.reply_queue.sync_q.put(result)

• We always treat it as if block=True now

• if self._write_connection is None:
• self._write_connection = self.connect(write=True)
• self.ds._prepare_connection(self._write_connection, self.name)

• return fn(self._write_connection)

  async def execute_fn(self, fn): - def in_thread(): - conn = getattr(connections, self.name, None) - if not conn: - conn = self.connect() - self.ds._prepare_connection(conn, self.name) - setattr(connections, self.name, conn) - return fn(conn) - - return await asyncio.get_event_loop().run_in_executor( - self.ds.executor, in_thread - ) + if self._read_connection is None: + self._read_connection = self.connect() + self.ds._prepare_connection(self._read_connection, self.name) + return fn(self._read_connection)

  async def execute( self, diff --git a/setup.py b/setup.py index 7f0562f..c41669c 100644 --- a/setup.py +++ b/setup.py @@ -44,20 +44,20 @@ setup( install_requires=[ "asgiref>=3.2.10,<3.6.0", "click>=7.1.1,<8.2.0", - "click-default-group~=1.2.2", + # "click-default-group~=1.2.2", "Jinja2>=2.10.3,<3.1.0", "hupper~=1.9", "httpx>=0.20", "pint~=0.9", "pluggy>=1.0,<1.1", - "uvicorn~=0.11", + # "uvicorn~=0.11", "aiofiles>=0.4,<0.9", "janus>=0.6.2,<1.1", "asgi-csrf>=0.9", "PyYAML>=5.3,<7.0", "mergedeep>=1.1.1,<1.4.0", "itsdangerous>=1.1,<3.0", - "python-baseconv==1.2.2", + # "python-baseconv==1.2.2", ], entry_points=""" [console_scripts] ```

This is actually mostly a documentation thing: here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time

In the code it's only used in these two places:

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_auth.py#L56-L60

I was going to vendor baseconv.py, but then I reconsidered - what if there are plugins out there that expect import baseconv to work because they have dependend on Datasette?

I used https://cs.github.com/ and as far as I can tell there aren't any!

So I'm going to remove that dependency and work out a smarter way to do this - probably by providing a utility function within Datasette itself.

I released a click-default-group-wheel package to solve that dependency issue. I've already upgraded sqlite-utils to that, so now you can use that in Pyodide:

• https://github.com/simonw/sqlite-utils/pull/429

python-baseconv is only used for actor cookie expiration times:

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20

Datasette never actually sets that cookie itself - it instead encourages plugins to set it in the authentication documentation here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time

I made a build of this branch and tested it like this: https://pyodide.org/en/stable/console.html

```pycon

import micropip await micropip.install("https://s3.amazonaws.com/simonwillison-cors-allowed-public/sqlite_utils-3.26-py3-none-any.whl") import sqlite_utils db = sqlite_utils.Database(memory=True) list(db.query("select 32443 + 55")) [{'32443 + 55': 32498}] ```

I'm going to ship this straight away as 3.26.1.

Also ran into this issue today using datasette package. The stack trace takes up my whole PowerShell history, though (recursionerror), but it also concerns the temporary directory. Our development machines have a very zealous scanner that appears to insert itself between every call to the filesystem. I suspected that was causing some racing, but this turned out not to be the case: inserting time.sleep(3) on line 451 of datasette/datasette/utils/__init__.py does not make the problem go away. Commenting out the tmp.cleanup() line does.

The next error I get is docker-specific, so that probably does resolve the Datasette error here.