html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/1981#issuecomment-1375142205,https://api.github.com/repos/simonw/datasette/issues/1981,1375142205,IC_kwDOBm6k_c5R9wE9,9599,2023-01-09T06:05:44Z,2023-01-09T06:05:44Z,OWNER,"I want the labels to take as much space as they need, but line up with each other.
If they are more than half the width of the page then the form fields should break to another line.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524983536,
https://github.com/simonw/datasette/issues/1980#issuecomment-1375062405,https://api.github.com/repos/simonw/datasette/issues/1980,1375062405,IC_kwDOBm6k_c5R9cmF,9599,2023-01-09T03:23:53Z,2023-01-09T03:23:53Z,OWNER,"Some potential solutions:
- Make it so the primary key is _always_ sortable - not a bad idea, the `sortable_columns` thing was designed to protect against sorting GIANT tables by a non-indexed column, and the primary key always has an index
- Make sure that `` menu never includes keys that cannot be sorted - this would be a good fix anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524867951,
https://github.com/simonw/datasette/issues/1980#issuecomment-1375061747,https://api.github.com/repos/simonw/datasette/issues/1980,1375061747,IC_kwDOBm6k_c5R9cbz,9599,2023-01-09T03:22:36Z,2023-01-09T03:22:36Z,OWNER,"The problem here is that the HTML ended up with this HTML in it:
```html
Sort...
Sort by id
```
But `id` is not a valid sort option any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524867951,
https://github.com/simonw/datasette/issues/1978#issuecomment-1374916593,https://api.github.com/repos/simonw/datasette/issues/1978,1374916593,IC_kwDOBm6k_c5R84_x,9599,2023-01-08T20:06:16Z,2023-01-08T20:06:16Z,OWNER,"Related issue - I'm not 100% settled on the design for these yet:
- https://github.com/simonw/datasette/issues/1048#issuecomment-1179757912","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1522778923,
https://github.com/simonw/datasette/issues/1979#issuecomment-1374659874,https://api.github.com/repos/simonw/datasette/issues/1979,1374659874,IC_kwDOBm6k_c5R76Ui,9599,2023-01-07T23:59:11Z,2023-01-07T23:59:11Z,OWNER,I back-ported this fix to `0.63.x` as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524076587,
https://github.com/simonw/datasette/issues/1979#issuecomment-1374659233,https://api.github.com/repos/simonw/datasette/issues/1979,1374659233,IC_kwDOBm6k_c5R76Kh,9599,2023-01-07T23:56:25Z,2023-01-07T23:56:25Z,OWNER,"```
% datasette --load-extension foo
Error: Your Python installation does not have the ability to load SQLite extensions.
More information: https://datasette.io/help/extensions
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524076587,
https://github.com/simonw/datasette/issues/1979#issuecomment-1374658468,https://api.github.com/repos/simonw/datasette/issues/1979,1374658468,IC_kwDOBm6k_c5R75-k,9599,2023-01-07T23:53:41Z,2023-01-07T23:53:41Z,OWNER,https://datasette.io/help/extensions now redirects to this new section of documentation: https://docs.datasette.io/en/latest/installation.html#a-note-about-extensions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524076587,
https://github.com/simonw/datasette/issues/1979#issuecomment-1374657204,https://api.github.com/repos/simonw/datasette/issues/1979,1374657204,IC_kwDOBm6k_c5R75q0,9599,2023-01-07T23:48:42Z,2023-01-07T23:48:42Z,OWNER,Once I ship the next release I should change that `/help/extensions` link to go to `/stable/` and not `/latest/`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524076587,
https://github.com/simonw/datasette/issues/1979#issuecomment-1374582375,https://api.github.com/repos/simonw/datasette/issues/1979,1374582375,IC_kwDOBm6k_c5R7nZn,9599,2023-01-07T19:22:39Z,2023-01-07T19:22:39Z,OWNER,"This helps:
```diff
diff --git a/datasette/cli.py b/datasette/cli.py
index 2b61292b..ea98879c 100644
--- a/datasette/cli.py
+++ b/datasette/cli.py
@@ -4,13 +4,15 @@ import click
from click import formatting
from click.types import CompositeParamType
from click_default_group import DefaultGroup
+import functools
import json
import os
import pathlib
import shutil
from subprocess import call
-import sys
from runpy import run_module
+import sys
+import textwrap
import webbrowser
from .app import (
OBSOLETE_SETTINGS,
@@ -126,7 +128,7 @@ class Setting(CompositeParamType):
def sqlite_extensions(fn):
- return click.option(
+ fn = click.option(
""sqlite_extensions"",
""--load-extension"",
type=LoadExtension(),
@@ -134,6 +136,25 @@ def sqlite_extensions(fn):
multiple=True,
help=""Path to a SQLite extension to load, and optional entrypoint"",
)(fn)
+ # Wrap it in a custom error handler
+ @functools.wraps(fn)
+ def wrapped(*args, **kwargs):
+ try:
+ return fn(*args, **kwargs)
+ except AttributeError as e:
+ if ""enable_load_extension"" in str(e):
+ raise click.ClickException(
+ textwrap.dedent(
+ """"""
+ Your Python installation does not have the ability to load SQLite extensions.
+
+ More information: https://docs.datasette.io/en/stable/installation.html#extensions
+ """"""
+ ).strip()
+ )
+ raise
+
+ return wrapped
@click.group(cls=DefaultGroup, default=""serve"", default_if_no_args=True)
```
Need to write help for that to link to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524076587,
https://github.com/simonw/datasette/issues/1409#issuecomment-1374555507,https://api.github.com/repos/simonw/datasette/issues/1409,1374555507,IC_kwDOBm6k_c5R7g1z,9599,2023-01-07T18:06:01Z,2023-01-07T18:06:01Z,OWNER,"Documentation:
- https://docs.datasette.io/en/latest/authentication.html#controlling-the-ability-to-execute-arbitrary-sql
- https://docs.datasette.io/en/latest/settings.html#setting-default-allow-sql
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,
https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1369333759,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1369333759,IC_kwDOCGYnMM5Rnl__,9599,2023-01-03T02:23:43Z,2023-01-03T02:23:43Z,OWNER,"The documentation here does at least say the following: https://sqlite-utils.datasette.io/en/3.30/python-api.html#reading-rows-from-a-file
> - **fp** (*BinaryIO*) -- a file-like object containing binary data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,
https://github.com/simonw/datasette/issues/1973#issuecomment-1368936146,https://api.github.com/repos/simonw/datasette/issues/1973,1368936146,IC_kwDOBm6k_c5RmE7S,9599,2023-01-02T13:13:03Z,2023-01-02T13:13:03Z,OWNER,It's quite handy that the label for foreign keys is made available to the custom render function here - if that was properly documented (and maybe had a slightly more intuitive API) I think it might be a good solution here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,
https://github.com/simonw/datasette/issues/1973#issuecomment-1368935320,https://api.github.com/repos/simonw/datasette/issues/1973,1368935320,IC_kwDOBm6k_c5RmEuY,9599,2023-01-02T13:12:00Z,2023-01-02T13:12:00Z,OWNER,"Thanks for catching this!
Definitely a bug. This might be a good opportunity to reconsider how his should work though.
It's possible that Datasette will eventually support alternative database backends - PostgreSQL, DuckDB and ClickHouse are all on my radar here.
If that happens, this plugin hook returning a SQLite Row wouldn't make sense.
Some kind of thing abstraction layer - maybe CustomRow, maybe something a little bit better - might be a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,
https://github.com/simonw/datasette/issues/782#issuecomment-1368285442,https://api.github.com/repos/simonw/datasette/issues/782,1368285442,IC_kwDOBm6k_c5RjmEC,9599,2022-12-31T22:02:16Z,2022-12-31T22:02:16Z,OWNER,"https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2 now returns `count`:
```json
{
""database"": ""fixtures"",
""table"": ""compound_three_primary_keys"",
""is_view"": false,
""human_description_en"": """",
""rows"": [
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""a"",
""content"": ""a-a-a""
},
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""b"",
""content"": ""a-a-b""
}
],
""truncated"": false,
""count"": 1001,
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/782#issuecomment-1368278278,https://api.github.com/repos/simonw/datasette/issues/782,1368278278,IC_kwDOBm6k_c5RjkUG,9599,2022-12-31T20:49:38Z,2022-12-31T20:49:38Z,OWNER,I'm going to rename `filtered_table_rows_count` to `count` - to match the SQL `count(*)` function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/782#issuecomment-1368269811,https://api.github.com/repos/simonw/datasette/issues/782,1368269811,IC_kwDOBm6k_c5RjiPz,9599,2022-12-31T19:33:09Z,2022-12-31T19:33:09Z,OWNER,Here's the so-far updated documentation for this change: https://github.com/simonw/datasette/blob/a2dca62360ad4a961d4c46f68eae41b7d5c7b2c9/docs/json_api.rst#different-shapes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/782#issuecomment-1368269283,https://api.github.com/repos/simonw/datasette/issues/782,1368269283,IC_kwDOBm6k_c5RjiHj,9599,2022-12-31T19:29:45Z,2022-12-31T19:29:45Z,OWNER,"https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2 now shows the new default:
```json
{
""database"": ""fixtures"",
""table"": ""compound_three_primary_keys"",
""is_view"": false,
""human_description_en"": """",
""rows"": [
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""a"",
""content"": ""a-a-a""
},
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""b"",
""content"": ""a-a-b""
}
],
```
The old format can be had like this: https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_shape=arrays","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/1970#issuecomment-1368268233,https://api.github.com/repos/simonw/datasette/issues/1970,1368268233,IC_kwDOBm6k_c5Rjh3J,9599,2022-12-31T19:22:58Z,2022-12-31T19:22:58Z,OWNER,"https://latest.datasette.io/_internal/databases shows the fix now:
1 ']]
FAILED tests/test_table_html.py::test_view_html - assert upper_content is None
FAILED tests/test_table_html.py::test_table_metadata - AssertionError: assert 'This <em&...t; is escaped' == 'simple_primary_key'
FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id\xa0▼', 'name']
FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk\xa0▼', 'name']
FAILED tests/test_table_html.py::test_column_metadata - AttributeError: 'NoneType' object has no attribute 'findAll'
======== 13 failed, 1279 passed, 3 skipped, 57 warnings in 572.40s (0:09:32) ========
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1355471341,https://api.github.com/repos/simonw/datasette/issues/1960,1355471341,IC_kwDOBm6k_c5Qytnt,9599,2022-12-16T19:23:33Z,2022-12-16T19:23:33Z,OWNER,"Trying this instead:
```
docker run -it ubuntu:22.04 /bin/bash
```
Then in that shell:
```
apt-get update
apt-get install python3.11 python3.11-venv git -y
cd /tmp
git clone https://github.com/simonw/datasette
cd datasette
git checkout async-tests
python3.11 -m venv venv
pip install -e '.[test]'
pytest
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1355445710,https://api.github.com/repos/simonw/datasette/issues/1960,1355445710,IC_kwDOBm6k_c5QynXO,9599,2022-12-16T19:06:49Z,2022-12-16T19:09:35Z,OWNER,"This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop.
https://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though.
https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1355325426,https://api.github.com/repos/simonw/datasette/issues/1960,1355325426,IC_kwDOBm6k_c5QyJ_y,9599,2022-12-16T18:00:40Z,2022-12-16T18:00:40Z,OWNER,Many of the failing tests pass on my laptop but fail in CI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1355319541,https://api.github.com/repos/simonw/datasette/issues/1960,1355319541,IC_kwDOBm6k_c5QyIj1,9599,2022-12-16T17:58:24Z,2022-12-16T17:58:46Z,OWNER,"> I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this.
It did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884
====== 11 failed, 1252 passed, 1 skipped, 1 warning in 185.77s (0:03:05) =======","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/issues/1959#issuecomment-1355317782,https://api.github.com/repos/simonw/datasette/issues/1959,1355317782,IC_kwDOBm6k_c5QyIIW,9599,2022-12-16T17:57:25Z,2022-12-16T17:57:25Z,OWNER,"Opened a follow-up issue here:
- #1962","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664,
https://github.com/simonw/datasette/pull/1960#issuecomment-1355317369,https://api.github.com/repos/simonw/datasette/issues/1960,1355317369,IC_kwDOBm6k_c5QyIB5,9599,2022-12-16T17:57:14Z,2022-12-16T17:57:14Z,OWNER,"Opened a follow-up issue here:
- #1962","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1355313058,https://api.github.com/repos/simonw/datasette/issues/1960,1355313058,IC_kwDOBm6k_c5QyG-i,9599,2022-12-16T17:53:17Z,2022-12-16T17:53:17Z,OWNER,"Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626
```
> assert response.json() == [{""id"": 1, ""name"": ""San Francisco""}]
E AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}]
E Full diff:
E - [{'id': 1, 'name': 'San Francisco'}]
E + {'error': 'Table not found: 𝐜𝐢𝐭𝐢𝐞𝐬', 'ok': False, 'status': 404, 'title': None}
```
A hunch: this failure suggests that maybe the fixtures tables were not correctly created when this test run. Maybe that can happen when `python -n auto` runs a bunch of separate processes and hence one of the tests randomly gets run in a fresh process and executes before the in-memory fixtures database has been fully populated.
I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1355300217,https://api.github.com/repos/simonw/datasette/issues/1960,1355300217,IC_kwDOBm6k_c5QyD15,9599,2022-12-16T17:44:55Z,2022-12-16T17:44:55Z,OWNER,That's enough for this round. I'll get the tests passing and land this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354269873,https://api.github.com/repos/simonw/datasette/issues/1960,1354269873,IC_kwDOBm6k_c5QuISx,9599,2022-12-16T06:11:43Z,2022-12-16T06:11:43Z,OWNER,"This is quite fast:
```
% pytest -m ds_client -n auto
================================================================================== test session starts ==================================================================================
platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0
SQLite: 3.39.4
rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0
asyncio: mode=strict
gw0 [291] / gw1 [291] / gw2 [291] / gw3 [291] / gw4 [291] / gw5 [291]
................................................................................................................................................................................. [ 60%]
.................................................................................................................. [100%]
================================================================================== 291 passed in 6.30s ==================================================================================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354148139,https://api.github.com/repos/simonw/datasette/issues/1960,1354148139,IC_kwDOBm6k_c5Qtqkr,9599,2022-12-16T03:32:52Z,2022-12-16T03:32:52Z,OWNER,"Got that done to: 68 passed in 14.92s
By implementing my own global variable - since `pytest` won't use a global fixture for me, I decided to do it for myself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354072344,https://api.github.com/repos/simonw/datasette/issues/1960,1354072344,IC_kwDOBm6k_c5QtYEY,9599,2022-12-16T02:00:07Z,2022-12-16T02:00:07Z,OWNER,"It did NOT speed it up:
68 passed in 26.26s","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354061440,https://api.github.com/repos/simonw/datasette/issues/1960,1354061440,IC_kwDOBm6k_c5QtVaA,9599,2022-12-16T01:45:38Z,2022-12-16T01:45:38Z,OWNER,"I'm going to do `test_table_html.py` next.
Currently: 68 passed in 17.20s
Will this speed it up?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354053151,https://api.github.com/repos/simonw/datasette/issues/1960,1354053151,IC_kwDOBm6k_c5QtTYf,9599,2022-12-16T01:33:22Z,2022-12-16T01:33:22Z,OWNER,"The thing with `Datasette(memory=False)` is tripping me up.
The problem is that the tests written against `app_client` - which I want to replace - all assume that there is no `_memory` database, because when you start Datasette with at least one database file it doesn't enable `_memory` unless you explicitly tell it to.
But the new `ds_client` fixture works by creating a named in-memory database called `fixtures`, which it does with a call to `ds.add_memory_database(""fixtures"")` after the object has been instantiated.
This results in a datasette instance that DOES have a `_memory` database, when we didn't want one.
My initial solution attempt was a huge hack - I decided that if you pass `memory=False` to the `Datasette` constructor it should mean ""don't add a `_memory` database even though I didn't pass any files"". I set a the default `memory` argument to `None`.
This is weird and surprising (`memory=False` no does something different from `memory=None`?) and I found other tests that it broke, like this one:
```python
def test_sql_errors_logged_to_stderr():
runner = CliRunner(mix_stderr=False)
result = runner.invoke(cli, [""--get"", ""/_memory.json?sql=select+blah""])
assert result.exit_code == 1
assert ""sql = 'select blah', params = {}: no such column: blah\n"" in result.stderr
```
It ended up with no `_memory` database because it turns out `datasette serve ...` passes `memory=False` without me realizing it.
So I'm going to undo that hack and teach the fixture to do this instead:
```python
db = ds.add_memory_database(""fixtures"")
ds.remove_database(""_memory"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354046627,https://api.github.com/repos/simonw/datasette/issues/1960,1354046627,IC_kwDOBm6k_c5QtRyj,9599,2022-12-16T01:23:21Z,2022-12-16T01:23:21Z,OWNER,"This does seem to help:
```diff
diff --git a/tests/conftest.py b/tests/conftest.py
index 1306c407..af9c7696 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = {
}
-@pytest.fixture(scope=""session"")
-def event_loop():
- return asyncio.get_event_loop()
-
-
-@pytest_asyncio.fixture(scope=""session"")
+@pytest_asyncio.fixture
async def ds_client():
from datasette.app import Datasette
from .fixtures import METADATA, PLUGINS_DIR
@@ -53,10 +48,11 @@ async def ds_client():
db = ds.add_memory_database(""fixtures"")
def prepare(conn):
- conn.executescript(TABLES)
- for sql, params in TABLE_PARAMETERIZED_SQL:
- with conn:
- conn.execute(sql, params)
+ if not conn.execute(""select count(*) from sqlite_master"").fetchone()[0]:
+ conn.executescript(TABLES)
+ for sql, params in TABLE_PARAMETERIZED_SQL:
+ with conn:
+ conn.execute(sql, params)
await db.execute_write_fn(prepare)
return ds.client
diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py
index 4f7bf08c..d588342c 100644
--- a/tests/plugins/my_plugin_2.py
+++ b/tests/plugins/my_plugin_2.py
@@ -117,7 +117,12 @@ def actor_from_request(datasette, request):
def permission_allowed(datasette, actor, action):
# Testing asyncio version of permission_allowed
async def inner():
- assert 2 == (await datasette.get_database().execute(""select 1 + 1"")).first()[0]
+ assert (
+ 2
+ == (
+ await datasette.get_database(""_internal"").execute(""select 1 + 1"")
+ ).first()[0]
+ )
if action == ""this_is_allowed_async"":
return True
elif action == ""this_is_denied_async"":
```
`pytest -m ds_client` now passes 134 tests.
Need to get `pytest -n auto` passing too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354036967,https://api.github.com/repos/simonw/datasette/issues/1960,1354036967,IC_kwDOBm6k_c5QtPbn,9599,2022-12-16T01:10:12Z,2022-12-16T01:10:12Z,OWNER,"If it does turn out that I can't use `scope=""session""` on this fixture it might not actually be a showstopper: I can take advantage of the fact that `memory_name=""...""` databases stay present in memory for the duration of the process, so I could have it such that each test that uses the `ds_client` fixture DOES construct a fresh `Datasette` instance, but doesn't need to populate the database since they can re-use the in-memory database from the previous object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/issues/1958#issuecomment-1354023960,https://api.github.com/repos/simonw/datasette/issues/1958,1354023960,IC_kwDOBm6k_c5QtMQY,9599,2022-12-16T00:58:14Z,2022-12-16T00:58:19Z,OWNER,"This is in the `0.63.x` branch now, ready to go out in a bug fix release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,
https://github.com/simonw/datasette/issues/1958#issuecomment-1354019543,https://api.github.com/repos/simonw/datasette/issues/1958,1354019543,IC_kwDOBm6k_c5QtLLX,9599,2022-12-16T00:54:11Z,2022-12-16T00:54:11Z,OWNER,"To test the fix, I did a fresh checkout of `simonw/datasette` and edited `Dockerfile` to look like this:
```dockerfile
FROM python:3.11.0-slim-bullseye as build
RUN apt-get update && \
apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg/info/*
RUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \
find /usr/local/lib -name '__pycache__' | xargs rm -r && \
rm -rf /root/.cache/pip
EXPOSE 8001
CMD [""datasette""]
```
Then I built it like this:
```
docker build -f Dockerfile -t datasette-pre-click .
```
And ran like this:
```
docker run -p 8001:8001 -v $(pwd):/mnt datasette-pre-click datasette --root -p 8001 -h 0.0.0.0
```
This exhibited the same problem.
Then I pushed a changed to branch and changed the line to:
```
RUN pip install https://github.com/simonw/datasette/archive/refs/heads/click-echo-root.zip && \
```
Ran this:
```
docker build -f Dockerfile -t datasette-post-click .
```
And this:
```
docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0
```
It fixed the problem!
```
datasette % docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0
http://0.0.0.0:8001/-/auth-token?token=6542dcf5c8f34f8d13f4af9ce728359c602469efb54029098562bd06c87ad26d
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,
https://github.com/simonw/datasette/issues/1958#issuecomment-1354008688,https://api.github.com/repos/simonw/datasette/issues/1958,1354008688,IC_kwDOBm6k_c5QtIhw,9599,2022-12-16T00:45:40Z,2022-12-16T00:45:40Z,OWNER,The fix may just be to switch to `click.echo()`. https://click.palletsprojects.com/en/8.1.x/api/#click.echo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,
https://github.com/simonw/datasette/issues/1958#issuecomment-1353977605,https://api.github.com/repos/simonw/datasette/issues/1958,1353977605,IC_kwDOBm6k_c5QtA8F,9599,2022-12-16T00:38:23Z,2022-12-16T00:38:23Z,OWNER,"Confirmed, I just got the same result:
```
- % docker run datasetteproject/datasette pip install datasette-upload-csvs
~ % docker commit $(docker ps -lq) datasette-with-plugins
sha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f
~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
^CINFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [1]
http://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98
```
Note how the auth-token URL is only displayed after you hit `Ctrl+C`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,
https://github.com/simonw/datasette/pull/1960#issuecomment-1353812913,https://api.github.com/repos/simonw/datasette/issues/1960,1353812913,IC_kwDOBm6k_c5QsYux,9599,2022-12-15T22:48:54Z,2022-12-15T22:48:54Z,OWNER,"This is all very broken:
```
% pytest -x --pdb
================================================================================== test session starts ==================================================================================
platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0
SQLite: 3.39.4
rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0
asyncio: mode=strict
collected 1295 items
tests/test_package.py .. [ 0%]
tests/test_cli.py . [ 0%]
tests/test_cli_serve_get.py .. [ 0%]
tests/test_cli.py . [ 0%]
tests/test_black.py . [ 0%]
tests/test_api.py E
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
fixturedef = , request = >
@pytest.hookimpl(hookwrapper=True)
def pytest_fixture_setup(
fixturedef: FixtureDef, request: SubRequest
) -> Optional[object]:
""""""Adjust the event loop policy when an event loop is produced.""""""
if fixturedef.argname == ""event_loop"":
outcome = yield
> loop = outcome.get_result()
/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pytest_asyncio/plugin.py:377:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: in event_loop
return asyncio.get_event_loop()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
def get_event_loop(self):
""""""Get the event loop for the current context.
Returns an instance of EventLoop or raises an exception.
""""""
if (self._local._loop is None and
not self._local._set_called and
threading.current_thread() is threading.main_thread()):
self.set_event_loop(self.new_event_loop())
if self._local._loop is None:
> raise RuntimeError('There is no current event loop in thread %r.'
% threading.current_thread().name)
E RuntimeError: There is no current event loop in thread 'MainThread'.
/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py:656: RuntimeError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py(656)get_event_loop()
-> raise RuntimeError('There is no current event loop in thread %r.'
(Pdb) q
=================================================================================== warnings summary ====================================================================================
tests/test_cli.py::test_inspect_cli_writes_to_file
tests/test_cli.py::test_inspect_cli
/Users/simon/Dropbox/Development/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop
loop = asyncio.get_event_loop()
tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error
tests/test_cli_serve_get.py::test_serve_with_get
/Users/simon/Dropbox/Development/datasette/datasette/cli.py:596: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(ds.invoke_startup())
tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error
tests/test_cli_serve_get.py::test_serve_with_get
/Users/simon/Dropbox/Development/datasette/datasette/cli.py:599: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(check_databases(ds))
tests/test_api.py::test_homepage
/Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: DeprecationWarning: There is no current event loop
return asyncio.get_event_loop()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================ short test summary info ================================================================================
ERROR tests/test_api.py::test_homepage - RuntimeError: There is no current event loop in thread 'MainThread'.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Quitting debugger !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================== 7 passed, 7 warnings, 1 error in 19.15s ========================================================================
(datasette) datasette %
```
The problem looks to be caused by this:
https://github.com/simonw/datasette/blob/87737aa1ace82fa7b54c60c41471ec9a661f5299/tests/conftest.py#L28-L30
Which I found necessary in order to have `async def` fixtures that could be shared on the `scope=""session""` basis.
Can I work around this, or is `scope=""session""` for async fixtures incompatible with my test suite for some reason?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1353805839,https://api.github.com/repos/simonw/datasette/issues/1960,1353805839,IC_kwDOBm6k_c5QsXAP,9599,2022-12-15T22:38:37Z,2022-12-15T22:38:37Z,OWNER,"I'm going to make `.status_code` work on `TestClient` response too, so I don't have to worry about using both `status` or `status_code` depending on which kind of object I am using.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1353765125,https://api.github.com/repos/simonw/datasette/issues/1960,1353765125,IC_kwDOBm6k_c5QsNEF,9599,2022-12-15T22:00:04Z,2022-12-15T22:00:04Z,OWNER,I'm going to punt on that for the moment and continue to use `app_client` for tests that use that mechanism.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1353763837,https://api.github.com/repos/simonw/datasette/issues/1960,1353763837,IC_kwDOBm6k_c5QsMv9,9599,2022-12-15T21:59:05Z,2022-12-15T21:59:05Z,OWNER,"Here's an annoying error:
```
> response4 = await ds_client.post(
""/-/logout"",
csrftoken_from=True,
cookies={""ds_actor"": ds_client.actor_cookie({""id"": ""test""})},
)
tests/test_auth.py:88:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = , path = '/-/logout'
kwargs = {'cookies': {'ds_actor': 'eyJhIjp7ImlkIjoidGVzdCJ9fQ.fuFCTJG5XE-RNnUM7dcnXx9sPvE'}, 'csrftoken_from': True}, client =
async def post(self, path, **kwargs):
await self.ds.invoke_startup()
async with httpx.AsyncClient(app=self.app) as client:
> return await client.post(self._fix(path), **kwargs)
E TypeError: AsyncClient.post() got an unexpected keyword argument 'csrftoken_from'
```
I need an alternative to the `csrftoken_from` mechanism I built for `TestClient`:
https://github.com/simonw/datasette/blob/0b68996cc511b3a801f0cd0157bd66332d75f46f/datasette/utils/testing.py#L77-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1353749401,https://api.github.com/repos/simonw/datasette/issues/1960,1353749401,IC_kwDOBm6k_c5QsJOZ,9599,2022-12-15T21:47:27Z,2022-12-15T21:47:27Z,OWNER,"I'm using this new mark:
```python
@pytest.mark.ds_client
```
Purely so I can run all of the tests that I've refactored using:
```
pytest -m ds_client
```
I'll likely remove this once the test refactoring project is complete.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/issues/1959#issuecomment-1353747370,https://api.github.com/repos/simonw/datasette/issues/1959,1353747370,IC_kwDOBm6k_c5QsIuq,9599,2022-12-15T21:45:14Z,2022-12-15T21:45:14Z,OWNER,I'm going to do this in a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664,
https://github.com/simonw/datasette/issues/1959#issuecomment-1353738075,https://api.github.com/repos/simonw/datasette/issues/1959,1353738075,IC_kwDOBm6k_c5QsGdb,9599,2022-12-15T21:35:56Z,2022-12-15T21:35:56Z,OWNER,"I built that `OldResponse` class:
```diff
diff --git a/tests/utils.py b/tests/utils.py
index 191ead9b..f39ac434 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -30,3 +30,25 @@ def inner_html(soup):
def has_load_extension():
conn = sqlite3.connect("":memory:"")
return hasattr(conn, ""enable_load_extension"")
+
+
+class OldResponse:
+ ""Transform an HTTPX response to simulate the older TestClient responses""
+ # https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091
+ def __init__(self, response):
+ self.response = response
+ self._json = None
+
+ @property
+ def headers(self):
+ return self.response.headers
+
+ @property
+ def status(self):
+ return self.response.status_code
+
+ @property
+ def json(self):
+ if self._json is None:
+ self._json = self.response.json()
+ return self._json
```
I can use it in tests like this:
```python
@pytest.mark.asyncio
async def test_homepage(ds_client):
response = OldResponse(await ds_client.get(""/.json""))
assert response.status == 200
assert ""application/json; charset=utf-8"" == response.headers[""content-type""]
assert response.json.keys() == {""fixtures"": 0}.keys()
d = response.json[""fixtures""]
assert d[""name""] == ""fixtures""
assert d[""tables_count""] == 24
assert len(d[""tables_and_views_truncated""]) == 5
assert d[""tables_and_views_more""] is True
# 4 hidden FTS tables + no_primary_key (hidden in metadata)
assert d[""hidden_tables_count""] == 6
# 201 in no_primary_key, plus 6 in other hidden tables:
assert d[""hidden_table_rows_sum""] == 207, response.json
assert d[""views_count""] == 4
```
But as I work through the tests I'm finding it's actually not too hard to port them over, so I likely won't use it after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664,
https://github.com/simonw/datasette/issues/1959#issuecomment-1353728682,https://api.github.com/repos/simonw/datasette/issues/1959,1353728682,IC_kwDOBm6k_c5QsEKq,9599,2022-12-15T21:28:35Z,2022-12-15T21:28:35Z,OWNER,"Got this error trying to have two tests use the same `ds_client` async fixture when I added `scope=""session""` to that fixture:
- https://github.com/tortoise/tortoise-orm/issues/638
Adding this to `conftest.py` (as suggested in that issue thread) seemed to fix it:
```python
@pytest.fixture(scope=""session"")
def event_loop():
return asyncio.get_event_loop()
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664,
https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091,https://api.github.com/repos/simonw/datasette/issues/1959,1353721091,IC_kwDOBm6k_c5QsCUD,9599,2022-12-15T21:20:32Z,2022-12-15T21:20:32Z,OWNER,Rather than tediously rewriting every single test to the new shape I'm going to try a wrapper for that HTTPX response that transforms it into an imitation of the one returned by the existing `TestClient` class.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664,
https://github.com/simonw/datasette/issues/1959#issuecomment-1353720559,https://api.github.com/repos/simonw/datasette/issues/1959,1353720559,IC_kwDOBm6k_c5QsCLv,9599,2022-12-15T21:19:56Z,2022-12-15T21:19:56Z,OWNER,"Here's a port of the first `def ...(app_client)` test. Note that the TestClient object works slightly differently from the HTTPX response returned by `await datasette.client.get(...)`:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index f3cb8876..b770b469 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -281,7 +281,7 @@ class Datasette:
raise
self.crossdb = crossdb
self.nolock = nolock
- if memory or crossdb or not self.files:
+ if memory or crossdb or (not self.files and memory is not False):
self.add_database(
Database(self, is_mutable=False, is_memory=True), name=""_memory""
)
diff --git a/pytest.ini b/pytest.ini
index 559e518c..0bcb0d1e 100644
--- a/pytest.ini
+++ b/pytest.ini
@@ -8,4 +8,5 @@ filterwarnings=
ignore:.*current_task.*:PendingDeprecationWarning
markers =
serial: tests to avoid using with pytest-xdist
+ ds_client: tests using the ds_client fixture
asyncio_mode = strict
diff --git a/tests/conftest.py b/tests/conftest.py
index cd735e12..648423ba 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -2,6 +2,7 @@ import httpx
import os
import pathlib
import pytest
+import pytest_asyncio
import re
import subprocess
import tempfile
@@ -23,6 +24,22 @@ UNDOCUMENTED_PERMISSIONS = {
}
+@pytest_asyncio.fixture
+async def ds_client():
+ from datasette.app import Datasette
+ from .fixtures import METADATA, PLUGINS_DIR
+ ds = Datasette(memory=False, metadata=METADATA, plugins_dir=PLUGINS_DIR)
+ from .fixtures import TABLES, TABLE_PARAMETERIZED_SQL
+ db = ds.add_memory_database(""fixtures"")
+ def prepare(conn):
+ conn.executescript(TABLES)
+ for sql, params in TABLE_PARAMETERIZED_SQL:
+ with conn:
+ conn.execute(sql, params)
+ await db.execute_write_fn(prepare)
+ return ds.client
+
+
def pytest_report_header(config):
return ""SQLite: {}"".format(
sqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchone()[0]
diff --git a/tests/test_api.py b/tests/test_api.py
index 5f2a6ea6..ddf4219c 100644
--- a/tests/test_api.py
+++ b/tests/test_api.py
@@ -23,12 +23,15 @@ import sys
import urllib
-def test_homepage(app_client):
- response = app_client.get(""/.json"")
- assert response.status == 200
+@pytest.mark.ds_client
+@pytest.mark.asyncio
+async def test_homepage(ds_client):
+ response = await ds_client.get(""/.json"")
+ assert response.status_code == 200
assert ""application/json; charset=utf-8"" == response.headers[""content-type""]
- assert response.json.keys() == {""fixtures"": 0}.keys()
- d = response.json[""fixtures""]
+ data = response.json()
+ assert data.keys() == {""fixtures"": 0}.keys()
+ d = data[""fixtures""]
assert d[""name""] == ""fixtures""
assert d[""tables_count""] == 24
assert len(d[""tables_and_views_truncated""]) == 5
@@ -36,7 +39,7 @@ def test_homepage(app_client):
# 4 hidden FTS tables + no_primary_key (hidden in metadata)
assert d[""hidden_tables_count""] == 6
# 201 in no_primary_key, plus 6 in other hidden tables:
- assert d[""hidden_table_rows_sum""] == 207, response.json
+ assert d[""hidden_table_rows_sum""] == 207, data
assert d[""views_count""] == 4
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664,
https://github.com/simonw/datasette/issues/1959#issuecomment-1353707828,https://api.github.com/repos/simonw/datasette/issues/1959,1353707828,IC_kwDOBm6k_c5Qr_E0,9599,2022-12-15T21:06:29Z,2022-12-15T21:06:29Z,OWNER,"Previous, abandoned attempt at this work (for #1843):
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 7e682498..cf35c3a2 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -228,7 +228,7 @@ class Datasette:
template_dir=None,
plugins_dir=None,
static_mounts=None,
- memory=False,
+ memory=None,
settings=None,
secret=None,
version_note=None,
@@ -238,6 +238,7 @@ class Datasette:
nolock=False,
):
self._startup_invoked = False
+ self._extra_on_startup = []
assert config_dir is None or isinstance(
config_dir, Path
), ""config_dir= should be a pathlib.Path""
@@ -278,7 +279,7 @@ class Datasette:
raise
self.crossdb = crossdb
self.nolock = nolock
- if memory or crossdb or not self.files:
+ if memory or crossdb or (not self.files and memory is not False):
self.add_database(
Database(self, is_mutable=False, is_memory=True), name=""_memory""
)
@@ -391,6 +392,9 @@ class Datasette:
self._root_token = secrets.token_hex(32)
self.client = DatasetteClient(self)
+ def _add_on_startup(self, fn):
+ self._extra_on_startup.append(fn)
+
async def refresh_schemas(self):
if self._refresh_schemas_lock.locked():
return
@@ -431,6 +435,8 @@ class Datasette:
# This must be called for Datasette to be in a usable state
if self._startup_invoked:
return
+ for fn in self._extra_on_startup:
+ await fn()
# Register permissions, but watch out for duplicate name/abbr
names = {}
abbrs = {}
@@ -1431,9 +1437,9 @@ class Datasette:
)
if self.setting(""trace_debug""):
asgi = AsgiTracer(asgi)
- asgi = AsgiRunOnFirstRequest(asgi, on_startup=[setup_db, self.invoke_startup])
for wrapper in pm.hook.asgi_wrapper(datasette=self):
asgi = wrapper(asgi)
+ asgi = AsgiRunOnFirstRequest(asgi, on_startup=[setup_db, self.invoke_startup])
return asgi
diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py
index 56690251..986755cb 100644
--- a/datasette/utils/asgi.py
+++ b/datasette/utils/asgi.py
@@ -423,9 +423,9 @@ class AsgiFileDownload:
class AsgiRunOnFirstRequest:
- def __init__(self, asgi, on_startup):
+ def __init__(self, app, on_startup):
assert isinstance(on_startup, list)
- self.asgi = asgi
+ self.app = app
self.on_startup = on_startup
self._started = False
@@ -434,4 +434,4 @@ class AsgiRunOnFirstRequest:
self._started = True
for hook in self.on_startup:
await hook()
- return await self.asgi(scope, receive, send)
+ return await self.app(scope, receive, send)
diff --git a/tests/conftest.py b/tests/conftest.py
index cd735e12..d1301943 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -23,6 +23,15 @@ UNDOCUMENTED_PERMISSIONS = {
}
+# @pytest.fixture(autouse=True)
+# def log_name_of_test_before_test(request):
+# # To help identify tests that are hanging
+# name = str(request.node)
+# with open(""/tmp/test.log"", ""a"") as f:
+# f.write(name + ""\n"")
+# yield
+
+
def pytest_report_header(config):
return ""SQLite: {}"".format(
sqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchone()[0]
diff --git a/tests/fixtures.py b/tests/fixtures.py
index a6700239..18d3f1b7 100644
--- a/tests/fixtures.py
+++ b/tests/fixtures.py
@@ -101,6 +101,19 @@ EXPECTED_PLUGINS = [
]
+def _populate_connection(conn):
+ # Drop any tables and views that exist
+ to_drop = conn.execute(
+ ""SELECT name, type FROM sqlite_master where type in ('table', 'view')""
+ ).fetchall()
+ for name, type in to_drop:
+ conn.execute(f""DROP {type} IF EXISTS [{name}]"")
+ conn.executescript(TABLES)
+ for sql, params in TABLE_PARAMETERIZED_SQL:
+ with conn:
+ conn.execute(sql, params)
+
+
@contextlib.contextmanager
def make_app_client(
sql_time_limit_ms=None,
@@ -117,45 +130,22 @@ def make_app_client(
metadata=None,
crossdb=False,
):
- with tempfile.TemporaryDirectory() as tmpdir:
- filepath = os.path.join(tmpdir, filename)
- if is_immutable:
- files = []
- immutables = [filepath]
- else:
- files = [filepath]
- immutables = []
- conn = sqlite3.connect(filepath)
- conn.executescript(TABLES)
- for sql, params in TABLE_PARAMETERIZED_SQL:
- with conn:
- conn.execute(sql, params)
- # Close the connection to avoid ""too many open files"" errors
- conn.close()
- if extra_databases is not None:
- for extra_filename, extra_sql in extra_databases.items():
- extra_filepath = os.path.join(tmpdir, extra_filename)
- c2 = sqlite3.connect(extra_filepath)
- c2.executescript(extra_sql)
- c2.close()
- # Insert at start to help test /-/databases ordering:
- files.insert(0, extra_filepath)
- os.chdir(os.path.dirname(filepath))
- settings = settings or {}
- for key, value in {
- ""default_page_size"": 50,
- ""max_returned_rows"": max_returned_rows or 100,
- ""sql_time_limit_ms"": sql_time_limit_ms or 200,
- # Default is 3 but this results in ""too many open files""
- # errors when running the full test suite:
- ""num_sql_threads"": 1,
- }.items():
- if key not in settings:
- settings[key] = value
+ settings = settings or {}
+ for key, value in {
+ ""default_page_size"": 50,
+ ""max_returned_rows"": max_returned_rows or 100,
+ ""sql_time_limit_ms"": sql_time_limit_ms or 200,
+ # Default is 3 but this results in ""too many open files""
+ # errors when running the full test suite:
+ ""num_sql_threads"": 1,
+ }.items():
+ if key not in settings:
+ settings[key] = value
+ # We can use an in-memory database, but only if we're not doing anything
+ # with is_immutable or extra_databases and filename is the default
+ if not is_immutable and not extra_databases and filename == ""fixtures.db"":
ds = Datasette(
- files,
- immutables=immutables,
- memory=memory,
+ memory=memory or False,
cors=cors,
metadata=metadata or METADATA,
plugins_dir=PLUGINS_DIR,
@@ -165,12 +155,57 @@ def make_app_client(
template_dir=template_dir,
crossdb=crossdb,
)
+ db = ds.add_memory_database(""fixtures"")
+
+ async def populate_fixtures():
+ print(""Here we go... populating fixtures"")
+ await db.execute_write_fn(_populate_connection)
+
+ ds._add_on_startup(populate_fixtures)
yield TestClient(ds)
- # Close as many database connections as possible
- # to try and avoid too many open files error
- for db in ds.databases.values():
- if not db.is_memory:
- db.close()
+ else:
+ with tempfile.TemporaryDirectory() as tmpdir:
+ filepath = os.path.join(tmpdir, filename)
+ if is_immutable:
+ files = []
+ immutables = [filepath]
+ else:
+ files = [filepath]
+ immutables = []
+
+ conn = sqlite3.connect(filepath)
+ _populate_connection(conn)
+ # Close the connection to reduce ""too many open files"" errors
+ conn.close()
+
+ if extra_databases is not None:
+ for extra_filename, extra_sql in extra_databases.items():
+ extra_filepath = os.path.join(tmpdir, extra_filename)
+ c2 = sqlite3.connect(extra_filepath)
+ c2.executescript(extra_sql)
+ c2.close()
+ # Insert at start to help test /-/databases ordering:
+ files.insert(0, extra_filepath)
+ os.chdir(os.path.dirname(filepath))
+ ds = Datasette(
+ files,
+ immutables=immutables,
+ memory=memory,
+ cors=cors,
+ metadata=metadata or METADATA,
+ plugins_dir=PLUGINS_DIR,
+ settings=settings,
+ inspect_data=inspect_data,
+ static_mounts=static_mounts,
+ template_dir=template_dir,
+ crossdb=crossdb,
+ )
+ yield TestClient(ds)
+ # Close as many database connections as possible
+ # to try and avoid too many open files error
+ for db in ds.databases.values():
+ if not db.is_memory:
+ db.close()
@pytest.fixture(scope=""session"")
diff --git a/tests/test_cli.py b/tests/test_cli.py
index d3e015fa..d9e4e457 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -1,5 +1,6 @@
from .fixtures import (
app_client,
+ app_client_with_cors,
make_app_client,
TestClient as _TestClient,
EXPECTED_PLUGINS,
@@ -38,7 +39,7 @@ def test_inspect_cli(app_client):
assert expected_count == database[""tables""][table_name][""count""]
-def test_inspect_cli_writes_to_file(app_client):
+def test_inspect_cli_writes_to_file(app_client_with_cors):
runner = CliRunner()
result = runner.invoke(
cli, [""inspect"", ""fixtures.db"", ""--inspect-file"", ""foo.json""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664,
https://github.com/simonw/datasette/issues/1959#issuecomment-1353705072,https://api.github.com/repos/simonw/datasette/issues/1959,1353705072,IC_kwDOBm6k_c5Qr-Zw,9599,2022-12-15T21:04:07Z,2022-12-15T21:04:07Z,OWNER,I'm going to start by getting every test that uses the raw `(app_client)` fixture and nothing else (194 at the moment) to switch to `async def` using a shared Datasette instance and `datasette.client.get()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664,
https://github.com/simonw/datasette/issues/1955#issuecomment-1353701674,https://api.github.com/repos/simonw/datasette/issues/1955,1353701674,IC_kwDOBm6k_c5Qr9kq,9599,2022-12-15T21:00:51Z,2022-12-15T21:00:51Z,OWNER,"OK, I've broken the test suite here.
I'm going to revert these two commits:
- https://github.com/simonw/datasette/commit/dc18f62089e5672d03176f217d7840cdafa5c447
- https://github.com/simonw/datasette/commit/51ee8caa4a697fa3f4120e93b1c205b714a6cdc7
Then I'll do a bunch of work making the test suite more robust before I try this again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622,