html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/pull/1960#issuecomment-1355823260,https://api.github.com/repos/simonw/datasette/issues/1960,1355823260,IC_kwDOBm6k_c5Q0Dic,9599,2022-12-16T23:36:07Z,2022-12-16T23:36:07Z,OWNER,"I ran `pytest --pdb -x` to drop into the debugger on the first failing test.
```
assert expected_rows == len(fetched)
> assert expected_pages == count
E assert 9 == 5
```
That's in `test_paginate_tables_and_views()`.
On a hunch, I checked the current settings:
```
(Pdb) ds_client.ds._settings
{'default_page_size': 50, 'max_returned_rows': 100, 'max_insert_rows': 100, 'num_sql_threads': 1, 'sql_time_limit_ms': 200, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'allow_facet': True, 'allow_download': True, 'allow_signed_tokens': True, 'max_signed_tokens_ttl': 0, 'suggest_facets': True, 'default_cache_ttl': 5, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'trace_debug': False, 'base_url': '/'}
```
Looks like something changed `default_page_size` to 50 and forgot to change it back!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/issues/1958#issuecomment-1353977605,https://api.github.com/repos/simonw/datasette/issues/1958,1353977605,IC_kwDOBm6k_c5QtA8F,9599,2022-12-16T00:38:23Z,2022-12-16T00:38:23Z,OWNER,"Confirmed, I just got the same result:
```
- % docker run datasetteproject/datasette pip install datasette-upload-csvs
~ % docker commit $(docker ps -lq) datasette-with-plugins
sha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f
~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
^CINFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [1]
http://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98
```
Note how the auth-token URL is only displayed after you hit `Ctrl+C`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,
https://github.com/simonw/datasette/issues/1958#issuecomment-1354008688,https://api.github.com/repos/simonw/datasette/issues/1958,1354008688,IC_kwDOBm6k_c5QtIhw,9599,2022-12-16T00:45:40Z,2022-12-16T00:45:40Z,OWNER,The fix may just be to switch to `click.echo()`. https://click.palletsprojects.com/en/8.1.x/api/#click.echo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,
https://github.com/simonw/datasette/issues/1958#issuecomment-1354019543,https://api.github.com/repos/simonw/datasette/issues/1958,1354019543,IC_kwDOBm6k_c5QtLLX,9599,2022-12-16T00:54:11Z,2022-12-16T00:54:11Z,OWNER,"To test the fix, I did a fresh checkout of `simonw/datasette` and edited `Dockerfile` to look like this:
```dockerfile
FROM python:3.11.0-slim-bullseye as build
RUN apt-get update && \
apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg/info/*
RUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \
find /usr/local/lib -name '__pycache__' | xargs rm -r && \
rm -rf /root/.cache/pip
EXPOSE 8001
CMD [""datasette""]
```
Then I built it like this:
```
docker build -f Dockerfile -t datasette-pre-click .
```
And ran like this:
```
docker run -p 8001:8001 -v $(pwd):/mnt datasette-pre-click datasette --root -p 8001 -h 0.0.0.0
```
This exhibited the same problem.
Then I pushed a changed to branch and changed the line to:
```
RUN pip install https://github.com/simonw/datasette/archive/refs/heads/click-echo-root.zip && \
```
Ran this:
```
docker build -f Dockerfile -t datasette-post-click .
```
And this:
```
docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0
```
It fixed the problem!
```
datasette % docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0
http://0.0.0.0:8001/-/auth-token?token=6542dcf5c8f34f8d13f4af9ce728359c602469efb54029098562bd06c87ad26d
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,
https://github.com/simonw/datasette/issues/1958#issuecomment-1354023960,https://api.github.com/repos/simonw/datasette/issues/1958,1354023960,IC_kwDOBm6k_c5QtMQY,9599,2022-12-16T00:58:14Z,2022-12-16T00:58:19Z,OWNER,"This is in the `0.63.x` branch now, ready to go out in a bug fix release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,
https://github.com/simonw/datasette/issues/1958#issuecomment-1354025319,https://api.github.com/repos/simonw/datasette/issues/1958,1354025319,IC_kwDOBm6k_c5QtMln,11729897,2022-12-16T00:59:12Z,2022-12-16T00:59:12Z,NONE,Awesome. Thank you @simonw.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354036967,https://api.github.com/repos/simonw/datasette/issues/1960,1354036967,IC_kwDOBm6k_c5QtPbn,9599,2022-12-16T01:10:12Z,2022-12-16T01:10:12Z,OWNER,"If it does turn out that I can't use `scope=""session""` on this fixture it might not actually be a showstopper: I can take advantage of the fact that `memory_name=""...""` databases stay present in memory for the duration of the process, so I could have it such that each test that uses the `ds_client` fixture DOES construct a fresh `Datasette` instance, but doesn't need to populate the database since they can re-use the in-memory database from the previous object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354046627,https://api.github.com/repos/simonw/datasette/issues/1960,1354046627,IC_kwDOBm6k_c5QtRyj,9599,2022-12-16T01:23:21Z,2022-12-16T01:23:21Z,OWNER,"This does seem to help:
```diff
diff --git a/tests/conftest.py b/tests/conftest.py
index 1306c407..af9c7696 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = {
}
-@pytest.fixture(scope=""session"")
-def event_loop():
- return asyncio.get_event_loop()
-
-
-@pytest_asyncio.fixture(scope=""session"")
+@pytest_asyncio.fixture
async def ds_client():
from datasette.app import Datasette
from .fixtures import METADATA, PLUGINS_DIR
@@ -53,10 +48,11 @@ async def ds_client():
db = ds.add_memory_database(""fixtures"")
def prepare(conn):
- conn.executescript(TABLES)
- for sql, params in TABLE_PARAMETERIZED_SQL:
- with conn:
- conn.execute(sql, params)
+ if not conn.execute(""select count(*) from sqlite_master"").fetchone()[0]:
+ conn.executescript(TABLES)
+ for sql, params in TABLE_PARAMETERIZED_SQL:
+ with conn:
+ conn.execute(sql, params)
await db.execute_write_fn(prepare)
return ds.client
diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py
index 4f7bf08c..d588342c 100644
--- a/tests/plugins/my_plugin_2.py
+++ b/tests/plugins/my_plugin_2.py
@@ -117,7 +117,12 @@ def actor_from_request(datasette, request):
def permission_allowed(datasette, actor, action):
# Testing asyncio version of permission_allowed
async def inner():
- assert 2 == (await datasette.get_database().execute(""select 1 + 1"")).first()[0]
+ assert (
+ 2
+ == (
+ await datasette.get_database(""_internal"").execute(""select 1 + 1"")
+ ).first()[0]
+ )
if action == ""this_is_allowed_async"":
return True
elif action == ""this_is_denied_async"":
```
`pytest -m ds_client` now passes 134 tests.
Need to get `pytest -n auto` passing too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354053151,https://api.github.com/repos/simonw/datasette/issues/1960,1354053151,IC_kwDOBm6k_c5QtTYf,9599,2022-12-16T01:33:22Z,2022-12-16T01:33:22Z,OWNER,"The thing with `Datasette(memory=False)` is tripping me up.
The problem is that the tests written against `app_client` - which I want to replace - all assume that there is no `_memory` database, because when you start Datasette with at least one database file it doesn't enable `_memory` unless you explicitly tell it to.
But the new `ds_client` fixture works by creating a named in-memory database called `fixtures`, which it does with a call to `ds.add_memory_database(""fixtures"")` after the object has been instantiated.
This results in a datasette instance that DOES have a `_memory` database, when we didn't want one.
My initial solution attempt was a huge hack - I decided that if you pass `memory=False` to the `Datasette` constructor it should mean ""don't add a `_memory` database even though I didn't pass any files"". I set a the default `memory` argument to `None`.
This is weird and surprising (`memory=False` no does something different from `memory=None`?) and I found other tests that it broke, like this one:
```python
def test_sql_errors_logged_to_stderr():
runner = CliRunner(mix_stderr=False)
result = runner.invoke(cli, [""--get"", ""/_memory.json?sql=select+blah""])
assert result.exit_code == 1
assert ""sql = 'select blah', params = {}: no such column: blah\n"" in result.stderr
```
It ended up with no `_memory` database because it turns out `datasette serve ...` passes `memory=False` without me realizing it.
So I'm going to undo that hack and teach the fixture to do this instead:
```python
db = ds.add_memory_database(""fixtures"")
ds.remove_database(""_memory"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354061440,https://api.github.com/repos/simonw/datasette/issues/1960,1354061440,IC_kwDOBm6k_c5QtVaA,9599,2022-12-16T01:45:38Z,2022-12-16T01:45:38Z,OWNER,"I'm going to do `test_table_html.py` next.
Currently: 68 passed in 17.20s
Will this speed it up?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,
https://github.com/simonw/datasette/pull/1960#issuecomment-1354062939,https://api.github.com/repos/simonw/datasette/issues/1960,1354062939,IC_kwDOBm6k_c5QtVxb,22429695,2022-12-16T01:48:06Z,2022-12-17T21:40:43Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Base: **92.19**% // Head: **92.05**% // Decreases project coverage by **`-0.13%`** :warning:
> Coverage data is based on head [(`770879a`)](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`0b68996`)](https://codecov.io/gh/simonw/datasette/commit/0b68996cc511b3a801f0cd0157bd66332d75f46f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> Patch coverage: 100.00% of modified lines in pull request are covered.
> :exclamation: Current head 770879a differs from pull request most recent head f42bca8. Consider uploading reports for the commit f42bca8 to get more accurate results
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #1960 +/- ##
==========================================
- Coverage 92.19% 92.05% -0.14%
==========================================
Files 38 38
Lines 5521 5527 +6
==========================================
- Hits 5090 5088 -2
- Misses 431 439 +8
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.13% <100.00%> (-0.57%)` | :arrow_down: |
| [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.83% <100.00%> (+0.24%)` | :arrow_up: |
| [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: |
| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.57% <0.00%> (-0.61%)` | :arrow_down: |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)