github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/issues/1479#issuecomment-1114601882 | https://api.github.com/repos/simonw/datasette/issues/1479 | 1114601882 | IC_kwDOBm6k_c5Cb3ma | 32839123 | 2022-05-02T08:10:27Z | 2022-05-02T11:54:49Z | NONE | Also ran into this issue today using `datasette package`. The stack trace takes up my whole PowerShell history, though (recursionerror), but it also concerns the temporary directory. Our development machines have a very zealous scanner that appears to insert itself between every call to the filesystem. I suspected that was causing some racing, but this turned out not to be the case: inserting `time.sleep(3)` on line 451 of `datasette/datasette/utils/__init__.py` does not make the problem go away. Commenting out the `tmp.cleanup()` line does. The next error I get is docker-specific, so that probably does resolve the Datasette error here. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1010112818 | |
https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115196863 | https://api.github.com/repos/simonw/sqlite-utils/issues/429 | 1115196863 | IC_kwDOCGYnMM5CeI2_ | 9599 | 2022-05-02T18:03:47Z | 2022-05-02T18:52:42Z | OWNER | I made a build of this branch and tested it like this: https://pyodide.org/en/stable/console.html ```pycon >>> import micropip >>> await micropip.install("https://s3.amazonaws.com/simonwillison-cors-allowed-public/sqlite_utils-3.26-py3-none-any.whl") >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> list(db.query("select 32443 + 55")) [{'32443 + 55': 32498}] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223177069 | |
https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115197644 | https://api.github.com/repos/simonw/sqlite-utils/issues/429 | 1115197644 | IC_kwDOCGYnMM5CeJDM | 9599 | 2022-05-02T18:04:28Z | 2022-05-02T18:04:28Z | OWNER | I'm going to ship this straight away as `3.26.1`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223177069 | |
https://github.com/simonw/datasette/issues/1733#issuecomment-1115256318 | https://api.github.com/repos/simonw/datasette/issues/1733 | 1115256318 | IC_kwDOBm6k_c5CeXX- | 9599 | 2022-05-02T19:05:55Z | 2022-05-02T19:05:55Z | OWNER | I released a `click-default-group-wheel` package to solve that dependency issue. I've already upgraded `sqlite-utils` to that, so now you can use that in Pyodide: - https://github.com/simonw/sqlite-utils/pull/429 `python-baseconv` is only used for actor cookie expiration times: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20 Datasette never actually sets that cookie itself - it instead encourages plugins to set it in the authentication documentation here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223234932 | |
https://github.com/simonw/datasette/issues/1733#issuecomment-1115258737 | https://api.github.com/repos/simonw/datasette/issues/1733 | 1115258737 | IC_kwDOBm6k_c5CeX9x | 9599 | 2022-05-02T19:08:17Z | 2022-05-02T19:08:17Z | OWNER | I was going to vendor `baseconv.py`, but then I reconsidered - what if there are plugins out there that expect `import baseconv` to work because they have dependend on Datasette? I used https://cs.github.com/ and as far as I can tell there aren't any! So I'm going to remove that dependency and work out a smarter way to do this - probably by providing a utility function within Datasette itself. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223234932 | |
https://github.com/simonw/datasette/issues/1733#issuecomment-1115262218 | https://api.github.com/repos/simonw/datasette/issues/1733 | 1115262218 | IC_kwDOBm6k_c5CeY0K | 9599 | 2022-05-02T19:11:51Z | 2022-05-02T19:14:01Z | OWNER | Here's the full diff I applied to Datasette to get it fully working in Pyodide: https://github.com/simonw/datasette/compare/94a3171b01fde5c52697aeeff052e3ad4bab5391...8af32bc5b03c30b1f7a4a8cc4bd80eb7e2ee7b81 And as a visible diff: ```diff diff --git a/datasette/app.py b/datasette/app.py index d269372..6c0c5fc 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -15,7 +15,6 @@ import pkg_resources import re import secrets import sys -import threading import traceback import urllib.parse from concurrent import futures @@ -26,7 +25,6 @@ from itsdangerous import URLSafeSerializer from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound -import uvicorn from .views.base import DatasetteError, ureg from .views.database import DatabaseDownload, DatabaseView @@ -813,7 +811,6 @@ class Datasette: }, "datasette": datasette_version, "asgi": "3.0", - "uvicorn": uvicorn.__version__, "sqlite": { "version": sqlite_version, "fts_versions": fts_versions, @@ -854,23 +851,7 @@ class Datasette: ] def _threads(self): - threads = list(threading.enumerate()) - d = { - "num_threads": len(threads), - "threads": [ - {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads - ], - } - # Only available in Python 3.7+ - if hasattr(asyncio, "all_tasks"): - tasks = asyncio.all_tasks() - d.update( - { - "num_tasks": len(tasks), - "tasks": [_cleaner_task_str(t) for t in tasks], - } - ) - return d + return {"num_threads": 0, "threads": []} def _actor(self, request): return {"actor": request.actor} diff --git a/datasette/database.py… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223234932 | |
https://github.com/simonw/datasette/issues/1733#issuecomment-1115268245 | https://api.github.com/repos/simonw/datasette/issues/1733 | 1115268245 | IC_kwDOBm6k_c5CeaSV | 9599 | 2022-05-02T19:18:11Z | 2022-05-02T19:18:11Z | OWNER | Maybe I can leave `uvicorn` as a dependency? Installing it works OK, it only generates errors when you try to import it: ```pycon Welcome to the Pyodide terminal emulator 🐍 Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM Type "help", "copyright", "credits" or "license" for more information. >>> import micropip >>> await micropip.install("uvicorn") >>> import uvicorn Traceback (most recent call last): File "<console>", line 1, in <module> File "/lib/python3.10/site-packages/uvicorn/__init__.py", line 1, in <module> from uvicorn.config import Config File "/lib/python3.10/site-packages/uvicorn/config.py", line 8, in <module> import ssl File "/lib/python3.10/ssl.py", line 98, in <module> import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl' >>> import ssl >>> import uvicorn Traceback (most recent call last): File "<console>", line 1, in <module> File "/lib/python3.10/site-packages/uvicorn/__init__.py", line 2, in <module> from uvicorn.main import Server, main, run File "/lib/python3.10/site-packages/uvicorn/main.py", line 24, in <module> from uvicorn.supervisors import ChangeReload, Multiprocess File "/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py", line 3, in <module> from uvicorn.supervisors.basereload import BaseReload File "/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py", line 12, in <module> from uvicorn.subprocess import get_subprocess File "/lib/python3.10/site-packages/uvicorn/subprocess.py", line 14, in <module> multiprocessing.allow_connection_pickling() File "/lib/python3.10/multiprocessing/context.py", line 170, in allow_connection_pickling from . import connection File "/lib/python3.10/multiprocessing/connection.py", line 21, in <module> import _multiprocessing ModuleNotFoundError: No module named '_multiprocessing' >>> import multiprocessing >>> import uvicorn Traceback (most recent call last): … | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223234932 | |
https://github.com/simonw/datasette/issues/1733#issuecomment-1115278325 | https://api.github.com/repos/simonw/datasette/issues/1733 | 1115278325 | IC_kwDOBm6k_c5Cecv1 | 9599 | 2022-05-02T19:29:05Z | 2022-05-02T19:29:05Z | OWNER | I'm going to add a Datasette setting to disable threading entirely, designed for usage in this particular case. I thought about adding a new setting, then I noticed this: datasette mydatabase.db --setting num_sql_threads 10 I'm going to let users set that to `0` to disable threaded execution of SQL queries. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223234932 | |
https://github.com/simonw/datasette/issues/1733#issuecomment-1115288284 | https://api.github.com/repos/simonw/datasette/issues/1733 | 1115288284 | IC_kwDOBm6k_c5CefLc | 9599 | 2022-05-02T19:40:33Z | 2022-05-02T19:40:33Z | OWNER | I'll release this as a `0.62a0` as soon as it's ready, so I can start testing it out in Pyodide for real. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223234932 | |
https://github.com/simonw/datasette/issues/1733#issuecomment-1115318303 | https://api.github.com/repos/simonw/datasette/issues/1733 | 1115318303 | IC_kwDOBm6k_c5Cemgf | 9599 | 2022-05-02T20:13:36Z | 2022-05-02T20:13:36Z | OWNER | I got a build from the `pyodide` branch to work! ``` Welcome to the Pyodide terminal emulator 🐍 Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM Type "help", "copyright", "credits" or "license" for more information. >>> import micropip >>> await micropip.install("https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl") Traceback (most recent call last): File "<console>", line 1, in <module> File "/lib/python3.10/asyncio/futures.py", line 284, in __await__ yield self # This tells Task to wait for completion. File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup future.result() File "/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception File "/lib/python3.10/asyncio/tasks.py", line 234, in __step result = coro.throw(exc) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 183, in install transaction = await self.gather_requirements(requirements, ctx, keep_going) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 173, in gather_requirements await gather(*requirement_promises) File "/lib/python3.10/asyncio/futures.py", line 284, in __await__ yield self # This tells Task to wait for completion. File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup future.result() File "/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception File "/lib/python3.10/asyncio/tasks.py", line 232, in __step result = coro.send(None) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 245, in add_requirement await self.add_wheel(name, wheel, version, (), ctx, transaction) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel await self.add_requirement(recurs_req, ctx, transaction) File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement await self.add_wheel( File "/lib/python3.10/site-packages/micropip/_micropip… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223234932 | |
https://github.com/simonw/datasette/issues/1733#issuecomment-1115318417 | https://api.github.com/repos/simonw/datasette/issues/1733 | 1115318417 | IC_kwDOBm6k_c5CemiR | 9599 | 2022-05-02T20:13:43Z | 2022-05-02T20:13:43Z | OWNER | This is good enough to push an alpha. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223234932 | |
https://github.com/simonw/datasette/issues/1733#issuecomment-1115404729 | https://api.github.com/repos/simonw/datasette/issues/1733 | 1115404729 | IC_kwDOBm6k_c5Ce7m5 | 9599 | 2022-05-02T21:49:01Z | 2022-05-02T21:49:38Z | OWNER | That alpha release works! https://pyodide.org/en/stable/console.html ```pycon Welcome to the Pyodide terminal emulator 🐍 Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM Type "help", "copyright", "credits" or "license" for more information. >>> import micropip >>> await micropip.install("datasette==0.62a0") >>> import ssl >>> import setuptools >>> from datasette.app import Datasette >>> ds = Datasette(memory=True, settings={"num_sql_threads": 0}) >>> await ds.client.get("/.json") <Response [200 OK]> >>> (await ds.client.get("/.json")).json() {'_memory': {'name': '_memory', 'hash': None, 'color': 'a6c7b9', 'path': '/_memory', 'tables_and_views_truncated': [], 'tab les_and_views_more': False, 'tables_count': 0, 'table_rows_sum': 0, 'show_table_row_counts': False, 'hidden_table_rows_sum' : 0, 'hidden_tables_count': 0, 'views_count': 0, 'private': False}} >>> ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223234932 | |
https://github.com/simonw/datasette/issues/1734#issuecomment-1115260999 | https://api.github.com/repos/simonw/datasette/issues/1734 | 1115260999 | IC_kwDOBm6k_c5CeYhH | 9599 | 2022-05-02T19:10:34Z | 2022-05-02T19:10:34Z | OWNER | This is actually mostly a documentation thing: here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time In the code it's only used in these two places: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20 https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_auth.py#L56-L60 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223241647 | |
https://github.com/simonw/datasette/issues/1734#issuecomment-1115282773 | https://api.github.com/repos/simonw/datasette/issues/1734 | 1115282773 | IC_kwDOBm6k_c5Ced1V | 9599 | 2022-05-02T19:34:15Z | 2022-05-02T19:34:15Z | OWNER | I'm going to vendor it and update the documentation. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223241647 | |
https://github.com/simonw/datasette/issues/1734#issuecomment-1115283922 | https://api.github.com/repos/simonw/datasette/issues/1734 | 1115283922 | IC_kwDOBm6k_c5CeeHS | 9599 | 2022-05-02T19:35:32Z | 2022-05-02T19:35:32Z | OWNER | I'll use my original from 2009: https://www.djangosnippets.org/snippets/1431/ | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223241647 | |
https://github.com/simonw/datasette/issues/1735#issuecomment-1115301733 | https://api.github.com/repos/simonw/datasette/issues/1735 | 1115301733 | IC_kwDOBm6k_c5Ceidl | 9599 | 2022-05-02T19:57:19Z | 2022-05-02T19:59:03Z | OWNER | This code breaks if that setting is 0: https://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/app.py#L291-L293 It's used here: https://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/database.py#L188-L190 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223263540 | |
https://github.com/simonw/datasette/issues/1737#issuecomment-1115462720 | https://api.github.com/repos/simonw/datasette/issues/1737 | 1115462720 | IC_kwDOBm6k_c5CfJxA | 9599 | 2022-05-02T23:25:03Z | 2022-05-02T23:25:03Z | OWNER | Here's a script that seems to work. It builds the wheel, starts a Python web server that serves the wheel, runs a test with `shot-scraper` and then shuts down the server again. ```bash #!/bin/bash # Build the wheel python3 -m build # Find name of wheel wheel=$(basename $(ls dist/*.whl)) # strip off the dist/ # Create a blank index page echo ' <script src="https://cdn.jsdelivr.net/pyodide/v0.20.0/full/pyodide.js"></script> ' > dist/index.html # Run a server for that dist/ folder cd dist python3 -m http.server 8529 & cd .. shot-scraper javascript http://localhost:8529/ " async () => { let pyodide = await loadPyodide(); await pyodide.loadPackage(['micropip', 'ssl', 'setuptools']); let output = await pyodide.runPythonAsync(\` import micropip await micropip.install('h11==0.12.0') await micropip.install('http://localhost:8529/$wheel') import ssl import setuptools from datasette.app import Datasette ds = Datasette(memory=True, settings={'num_sql_threads': 0}) (await ds.client.get('/_memory.json?sql=select+55+as+itworks&_shape=array')).text \`); if (JSON.parse(output)[0].itworks != 55) { throw 'Got ' + output + ', expected itworks: 55'; } return 'Test passed!'; } " # Shut down the server pkill -f 'http.server 8529' ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223459734 | |
https://github.com/simonw/datasette/issues/1737#issuecomment-1115464097 | https://api.github.com/repos/simonw/datasette/issues/1737 | 1115464097 | IC_kwDOBm6k_c5CfKGh | 9599 | 2022-05-02T23:27:40Z | 2022-05-02T23:27:40Z | OWNER | I'm going to start off by running this manually - I may run it on every commit once this is all a little bit more stable. I can base the workflow on https://github.com/simonw/scrape-hacker-news-by-domain/blob/main/.github/workflows/scrape.yml | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223459734 | |
https://github.com/simonw/datasette/issues/1737#issuecomment-1115468193 | https://api.github.com/repos/simonw/datasette/issues/1737 | 1115468193 | IC_kwDOBm6k_c5CfLGh | 9599 | 2022-05-02T23:35:26Z | 2022-05-02T23:35:26Z | OWNER | https://github.com/simonw/datasette/runs/6265915080?check_suite_focus=true failed but looks like it passed because I forgot to use `set -e` at the start of the bash script. It failed because it didn't have `build` available. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223459734 | |
https://github.com/simonw/datasette/issues/1737#issuecomment-1115470180 | https://api.github.com/repos/simonw/datasette/issues/1737 | 1115470180 | IC_kwDOBm6k_c5CfLlk | 9599 | 2022-05-02T23:39:29Z | 2022-05-02T23:39:29Z | OWNER | Test ran in 38 seconds and passed! https://github.com/simonw/datasette/runs/6265954274?check_suite_focus=true I'm going to have it run on every commit and PR. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1223459734 |