issue_comments
8,883 rows where user = 9599 sorted by updated_at descending
This data as json, CSV (advanced)
user 1
- simonw · 8,065 ✖
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
1375142205 | https://github.com/simonw/datasette/issues/1981#issuecomment-1375142205 | https://api.github.com/repos/simonw/datasette/issues/1981 | IC_kwDOBm6k_c5R9wE9 | simonw 9599 | 2023-01-09T06:05:44Z | 2023-01-09T06:05:44Z | OWNER | I want the labels to take as much space as they need, but line up with each other. If they are more than half the width of the page then the form fields should break to another line. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Canned query field labels truncated 1524983536 | |
1375062405 | https://github.com/simonw/datasette/issues/1980#issuecomment-1375062405 | https://api.github.com/repos/simonw/datasette/issues/1980 | IC_kwDOBm6k_c5R9cmF | simonw 9599 | 2023-01-09T03:23:53Z | 2023-01-09T03:23:53Z | OWNER | Some potential solutions:
- Make it so the primary key is always sortable - not a bad idea, the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"Cannot sort table by id" when sortable_columns is used 1524867951 | |
1375061747 | https://github.com/simonw/datasette/issues/1980#issuecomment-1375061747 | https://api.github.com/repos/simonw/datasette/issues/1980 | IC_kwDOBm6k_c5R9cbz | simonw 9599 | 2023-01-09T03:22:36Z | 2023-01-09T03:22:36Z | OWNER | The problem here is that the HTML ended up with this HTML in it:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"Cannot sort table by id" when sortable_columns is used 1524867951 | |
1374916593 | https://github.com/simonw/datasette/issues/1978#issuecomment-1374916593 | https://api.github.com/repos/simonw/datasette/issues/1978 | IC_kwDOBm6k_c5R84_x | simonw 9599 | 2023-01-08T20:06:16Z | 2023-01-08T20:06:16Z | OWNER | Related issue - I'm not 100% settled on the design for these yet: - https://github.com/simonw/datasette/issues/1048#issuecomment-1179757912 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Document datasette.urls.row and row_blob 1522778923 | |
1374659874 | https://github.com/simonw/datasette/issues/1979#issuecomment-1374659874 | https://api.github.com/repos/simonw/datasette/issues/1979 | IC_kwDOBm6k_c5R76Ui | simonw 9599 | 2023-01-07T23:59:11Z | 2023-01-07T23:59:11Z | OWNER | I back-ported this fix to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
More useful error message if enable_load_extension is not available 1524076587 | |
1374659233 | https://github.com/simonw/datasette/issues/1979#issuecomment-1374659233 | https://api.github.com/repos/simonw/datasette/issues/1979 | IC_kwDOBm6k_c5R76Kh | simonw 9599 | 2023-01-07T23:56:25Z | 2023-01-07T23:56:25Z | OWNER | ```
% datasette --load-extension foo More information: https://datasette.io/help/extensions ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
More useful error message if enable_load_extension is not available 1524076587 | |
1374658468 | https://github.com/simonw/datasette/issues/1979#issuecomment-1374658468 | https://api.github.com/repos/simonw/datasette/issues/1979 | IC_kwDOBm6k_c5R75-k | simonw 9599 | 2023-01-07T23:53:41Z | 2023-01-07T23:53:41Z | OWNER | https://datasette.io/help/extensions now redirects to this new section of documentation: https://docs.datasette.io/en/latest/installation.html#a-note-about-extensions |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
More useful error message if enable_load_extension is not available 1524076587 | |
1374657204 | https://github.com/simonw/datasette/issues/1979#issuecomment-1374657204 | https://api.github.com/repos/simonw/datasette/issues/1979 | IC_kwDOBm6k_c5R75q0 | simonw 9599 | 2023-01-07T23:48:42Z | 2023-01-07T23:48:42Z | OWNER | Once I ship the next release I should change that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
More useful error message if enable_load_extension is not available 1524076587 | |
1374582375 | https://github.com/simonw/datasette/issues/1979#issuecomment-1374582375 | https://api.github.com/repos/simonw/datasette/issues/1979 | IC_kwDOBm6k_c5R7nZn | simonw 9599 | 2023-01-07T19:22:39Z | 2023-01-07T19:22:39Z | OWNER | This helps: ```diff diff --git a/datasette/cli.py b/datasette/cli.py index 2b61292b..ea98879c 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -4,13 +4,15 @@ import click from click import formatting from click.types import CompositeParamType from click_default_group import DefaultGroup +import functools import json import os import pathlib import shutil from subprocess import call -import sys from runpy import run_module +import sys +import textwrap import webbrowser from .app import ( OBSOLETE_SETTINGS, @@ -126,7 +128,7 @@ class Setting(CompositeParamType): def sqlite_extensions(fn): - return click.option( + fn = click.option( "sqlite_extensions", "--load-extension", type=LoadExtension(), @@ -134,6 +136,25 @@ def sqlite_extensions(fn): multiple=True, help="Path to a SQLite extension to load, and optional entrypoint", )(fn) + # Wrap it in a custom error handler + @functools.wraps(fn) + def wrapped(args, kwargs): + try: + return fn(args, **kwargs) + except AttributeError as e: + if "enable_load_extension" in str(e): + raise click.ClickException( + textwrap.dedent( + """ + Your Python installation does not have the ability to load SQLite extensions. + + More information: https://docs.datasette.io/en/stable/installation.html#extensions + """ + ).strip() + ) + raise + + return wrapped @click.group(cls=DefaultGroup, default="serve", default_if_no_args=True) ``` Need to write help for that to link to. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
More useful error message if enable_load_extension is not available 1524076587 | |
1374555507 | https://github.com/simonw/datasette/issues/1409#issuecomment-1374555507 | https://api.github.com/repos/simonw/datasette/issues/1409 | IC_kwDOBm6k_c5R7g1z | simonw 9599 | 2023-01-07T18:06:01Z | 2023-01-07T18:06:01Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278 | ||
1369333759 | https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1369333759 | https://api.github.com/repos/simonw/sqlite-utils/issues/520 | IC_kwDOCGYnMM5Rnl__ | simonw 9599 | 2023-01-03T02:23:43Z | 2023-01-03T02:23:43Z | OWNER | The documentation here does at least say the following: https://sqlite-utils.datasette.io/en/3.30/python-api.html#reading-rows-from-a-file
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
rows_from_file() raises confusing error if file-like object is not in binary mode 1516644980 | |
1368936146 | https://github.com/simonw/datasette/issues/1973#issuecomment-1368936146 | https://api.github.com/repos/simonw/datasette/issues/1973 | IC_kwDOBm6k_c5RmE7S | simonw 9599 | 2023-01-02T13:13:03Z | 2023-01-02T13:13:03Z | OWNER | It's quite handy that the label for foreign keys is made available to the custom render function here - if that was properly documented (and maybe had a slightly more intuitive API) I think it might be a good solution here. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
render_cell plugin hook's row object is not a sqlite.Row 1515815014 | |
1368935320 | https://github.com/simonw/datasette/issues/1973#issuecomment-1368935320 | https://api.github.com/repos/simonw/datasette/issues/1973 | IC_kwDOBm6k_c5RmEuY | simonw 9599 | 2023-01-02T13:12:00Z | 2023-01-02T13:12:00Z | OWNER | Thanks for catching this! Definitely a bug. This might be a good opportunity to reconsider how his should work though. It's possible that Datasette will eventually support alternative database backends - PostgreSQL, DuckDB and ClickHouse are all on my radar here. If that happens, this plugin hook returning a SQLite Row wouldn't make sense. Some kind of thing abstraction layer - maybe CustomRow, maybe something a little bit better - might be a good idea. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
render_cell plugin hook's row object is not a sqlite.Row 1515815014 | |
1368285442 | https://github.com/simonw/datasette/issues/782#issuecomment-1368285442 | https://api.github.com/repos/simonw/datasette/issues/782 | IC_kwDOBm6k_c5RjmEC | simonw 9599 | 2022-12-31T22:02:16Z | 2022-12-31T22:02:16Z | OWNER | https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2 now returns
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
1368278278 | https://github.com/simonw/datasette/issues/782#issuecomment-1368278278 | https://api.github.com/repos/simonw/datasette/issues/782 | IC_kwDOBm6k_c5RjkUG | simonw 9599 | 2022-12-31T20:49:38Z | 2022-12-31T20:49:38Z | OWNER | I'm going to rename |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
1368269811 | https://github.com/simonw/datasette/issues/782#issuecomment-1368269811 | https://api.github.com/repos/simonw/datasette/issues/782 | IC_kwDOBm6k_c5RjiPz | simonw 9599 | 2022-12-31T19:33:09Z | 2022-12-31T19:33:09Z | OWNER | Here's the so-far updated documentation for this change: https://github.com/simonw/datasette/blob/a2dca62360ad4a961d4c46f68eae41b7d5c7b2c9/docs/json_api.rst#different-shapes |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
1368269283 | https://github.com/simonw/datasette/issues/782#issuecomment-1368269283 | https://api.github.com/repos/simonw/datasette/issues/782 | IC_kwDOBm6k_c5RjiHj | simonw 9599 | 2022-12-31T19:29:45Z | 2022-12-31T19:29:45Z | OWNER | https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2 now shows the new default:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
1368268233 | https://github.com/simonw/datasette/issues/1970#issuecomment-1368268233 | https://api.github.com/repos/simonw/datasette/issues/1970 | IC_kwDOBm6k_c5Rjh3J | simonw 9599 | 2022-12-31T19:22:58Z | 2022-12-31T19:22:58Z | OWNER | https://latest.datasette.io/_internal/databases shows the fix now: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Path "None" in _internal database table 1515182998 | |
1368268148 | https://github.com/simonw/datasette/issues/782#issuecomment-1368268148 | https://api.github.com/repos/simonw/datasette/issues/782 | IC_kwDOBm6k_c5Rjh10 | simonw 9599 | 2022-12-31T19:22:07Z | 2022-12-31T19:22:07Z | OWNER | It turned out the most significant part of this change - switching from an array of arrays to an array of objects for the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
1368267484 | https://github.com/simonw/datasette/pull/1967#issuecomment-1368267484 | https://api.github.com/repos/simonw/datasette/issues/1967 | IC_kwDOBm6k_c5Rjhrc | simonw 9599 | 2022-12-31T19:15:50Z | 2022-12-31T19:15:50Z | OWNER | My Firefox tab before: And after: |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
Add favicon to documentation 1503010009 | |
1368267406 | https://github.com/simonw/datasette/issues/1966#issuecomment-1368267406 | https://api.github.com/repos/simonw/datasette/issues/1966 | IC_kwDOBm6k_c5RjhqO | simonw 9599 | 2022-12-31T19:15:10Z | 2022-12-31T19:15:10Z | OWNER | It's working again now - not sure what happened there, I was paying for Heroku for that already. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Broken link to live demo in Getting started docs 1501900064 | |
1368266904 | https://github.com/simonw/datasette/issues/1971#issuecomment-1368266904 | https://api.github.com/repos/simonw/datasette/issues/1971 | IC_kwDOBm6k_c5RjhiY | simonw 9599 | 2022-12-31T19:09:48Z | 2022-12-31T19:09:48Z | OWNER | For the moment I'm going to pin to Sphinx 5.3.0 Note that I got this warning there:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Upgrade for Sphinx 6.0 (once Furo has support for it) 1515185383 | |
1368266658 | https://github.com/simonw/datasette/issues/1971#issuecomment-1368266658 | https://api.github.com/repos/simonw/datasette/issues/1971 | IC_kwDOBm6k_c5Rjhei | simonw 9599 | 2022-12-31T19:07:19Z | 2022-12-31T19:07:19Z | OWNER | So it looks like Furo doesn't work with Sphinx 6.0 yet - there' s a fix on |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Upgrade for Sphinx 6.0 (once Furo has support for it) 1515185383 | |
1368266457 | https://github.com/simonw/datasette/issues/1971#issuecomment-1368266457 | https://api.github.com/repos/simonw/datasette/issues/1971 | IC_kwDOBm6k_c5RjhbZ | simonw 9599 | 2022-12-31T19:05:26Z | 2022-12-31T19:05:26Z | OWNER | Ran this locally:
Got these warnings:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Upgrade for Sphinx 6.0 (once Furo has support for it) 1515185383 | |
1368265354 | https://github.com/simonw/datasette/pull/1967#issuecomment-1368265354 | https://api.github.com/repos/simonw/datasette/issues/1967 | IC_kwDOBm6k_c5RjhKK | simonw 9599 | 2022-12-31T18:56:35Z | 2022-12-31T18:56:35Z | OWNER | Thanks for this! Datasette has a favicon in this file: https://github.com/simonw/datasette/blob/2b4a8aa0833068da25072b6247bf349949ca00ee/datasette/static/favicon.png - so I'll copy that over and use it here as well. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add favicon to documentation 1503010009 | |
1368264882 | https://github.com/simonw/datasette/issues/1970#issuecomment-1368264882 | https://api.github.com/repos/simonw/datasette/issues/1970 | IC_kwDOBm6k_c5RjhCy | simonw 9599 | 2022-12-31T18:52:01Z | 2022-12-31T18:52:01Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Path "None" in _internal database table 1515182998 | ||
1356827218 | https://github.com/simonw/datasette/pull/1965#issuecomment-1356827218 | https://api.github.com/repos/simonw/datasette/issues/1965 | IC_kwDOBm6k_c5Q34pS | simonw 9599 | 2022-12-18T16:01:36Z | 2022-12-18T16:01:36Z | OWNER | Will link to this from my TIL shortly. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Detect server start/stop more reliably. 1501843596 | |
1356827167 | https://github.com/simonw/datasette/pull/1965#issuecomment-1356827167 | https://api.github.com/repos/simonw/datasette/issues/1965 | IC_kwDOBm6k_c5Q34of | simonw 9599 | 2022-12-18T16:01:22Z | 2022-12-18T16:01:22Z | OWNER | This is great, thank you! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Detect server start/stop more reliably. 1501843596 | |
1356697705 | https://github.com/simonw/datasette/issues/1964#issuecomment-1356697705 | https://api.github.com/repos/simonw/datasette/issues/1964 | IC_kwDOBm6k_c5Q3ZBp | simonw 9599 | 2022-12-18T06:37:23Z | 2022-12-18T06:37:23Z | OWNER | I'm certain the two other cog menus (the app menu on the right of the nav bar and the column action menus) have the same problem. Would be great to figure out the right ARIA attributes for these too. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Cog menu is not keyboard accessible (also no ARIA) 1501778647 | |
1356694671 | https://github.com/simonw/datasette/issues/1771#issuecomment-1356694671 | https://api.github.com/repos/simonw/datasette/issues/1771 | IC_kwDOBm6k_c5Q3YSP | simonw 9599 | 2022-12-18T06:34:20Z | 2022-12-18T06:34:20Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
minor a11y: <select> has no visual indicator when tabbed to 1306984363 | ||
1356680769 | https://github.com/simonw/datasette/issues/1771#issuecomment-1356680769 | https://api.github.com/repos/simonw/datasette/issues/1771 | IC_kwDOBm6k_c5Q3U5B | simonw 9599 | 2022-12-18T05:56:05Z | 2022-12-18T05:56:05Z | OWNER | This does the trick:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
minor a11y: <select> has no visual indicator when tabbed to 1306984363 | |
1356655630 | https://github.com/simonw/datasette/issues/1771#issuecomment-1356655630 | https://api.github.com/repos/simonw/datasette/issues/1771 | IC_kwDOBm6k_c5Q3OwO | simonw 9599 | 2022-12-18T03:43:12Z | 2022-12-18T03:43:12Z | OWNER | The border is actually on the div that wraps the select box: I tried adding a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
minor a11y: <select> has no visual indicator when tabbed to 1306984363 | |
1356655217 | https://github.com/simonw/datasette/issues/1771#issuecomment-1356655217 | https://api.github.com/repos/simonw/datasette/issues/1771 | IC_kwDOBm6k_c5Q3Opx | simonw 9599 | 2022-12-18T03:38:16Z | 2022-12-18T03:38:16Z | OWNER | OK I see what you mean: https://latest.datasette.io/fixtures/attraction_characteristic |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
minor a11y: <select> has no visual indicator when tabbed to 1306984363 | |
1356652057 | https://github.com/simonw/datasette/issues/1963#issuecomment-1356652057 | https://api.github.com/repos/simonw/datasette/issues/1963 | IC_kwDOBm6k_c5Q3N4Z | simonw 9599 | 2022-12-18T03:23:22Z | 2022-12-18T03:23:22Z | OWNER | https://pypi.org/project/datasette/0.63.3/ is released. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0.63.3 bugfix release 1501713288 | |
1356651943 | https://github.com/simonw/datasette/issues/1963#issuecomment-1356651943 | https://api.github.com/repos/simonw/datasette/issues/1963 | IC_kwDOBm6k_c5Q3N2n | simonw 9599 | 2022-12-18T03:23:03Z | 2022-12-18T03:23:03Z | OWNER | Oh that's annoying... every step in publish succeeded except the static docs one: https://github.com/simonw/datasette/actions/runs/3723015082/jobs/6314292722 This means the documentation database used to update the search engine on https://datasette.io/ won't reflect the very latest changelog. I'm OK with that - I'll fix this workflow so that next time I publish a release this will work correctly. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0.63.3 bugfix release 1501713288 | |
1356640463 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356640463 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3LDP | simonw 9599 | 2022-12-18T02:45:18Z | 2022-12-18T02:45:18Z | OWNER | ... and with this change, the following now works correctly:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356640266 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356640266 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3LAK | simonw 9599 | 2022-12-18T02:43:00Z | 2022-12-18T02:43:00Z | OWNER | https://github.com/simonw/datasette/actions/runs/3722908296/jobs/6314093163 shows that new test passing in CI: ``` Generated a certificate for 'localhost', '127.0.0.1', '::1' Configure your server to use the following files: cert=/home/runner/work/datasette/datasette/server.pem key=/home/runner/work/datasette/datasette/server.key Configure your client to use the following files: cert=/home/runner/work/datasette/datasette/client.pem INFO: Started server process [4036] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on https://127.0.0.1:8152/ (Press CTRL+C to quit) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed INFO: 127.0.0.1:56726 - "GET /_memory.json HTTP/1.1" 200 OK 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 213 0 213 0 0 11542 0 --:--:-- --:--:-- --:--:-- 11833 INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [4036] {"database": "_memory", "private": false, "path": "/_memory", "size": 0, "tables": [], "hidden_count": 0, "views": [], "queries": [], "allow_execute_sql": true, "table_columns": {}, "query_ms": 1.4545189999921604}0 ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356639873 | https://github.com/simonw/datasette/issues/1221#issuecomment-1356639873 | https://api.github.com/repos/simonw/datasette/issues/1221 | IC_kwDOBm6k_c5Q3K6B | simonw 9599 | 2022-12-18T02:39:04Z | 2022-12-18T02:39:04Z | OWNER | I ended up moving this test out of Python and into a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support SSL/TLS directly 806849424 | |
1356633937 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356633937 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3JdR | simonw 9599 | 2022-12-18T02:28:22Z | 2022-12-18T02:28:22Z | OWNER | This !/bin/bashGenerate certificatespython -m trustme This creates server.pem, server.key, client.pemStart the server in the backgrounddatasette --memory \ --ssl-keyfile=server.key \ --ssl-certfile=server.pem \ -p 8152 & Store the background process ID in a variableserver_pid=$! Wait for the server to startsleep 2 Make a test request using curlcurl -f --cacert client.pem 'https://localhost:8152/_memory.json' Save curl's exit code (-f option causes it to return one on HTTP errors)curl_exit_code=$? Shut down the serverkill $server_pid sleep 1 Clean up the certificatesrm server.pem server.key client.pem echo $curl_exit_code exit $curl_exit_code ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356630092 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356630092 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3IhM | simonw 9599 | 2022-12-18T02:20:01Z | 2022-12-18T02:20:01Z | OWNER | Asked ChatGPT:
It gave me: ```bash !/bin/bashStart the server in the backgrounddatasette -p 8002 & Store the background process ID in a variableserver_pid=$! Make a test request using curlcurl http://localhost:8002 Shut down the serverkill $server_pid ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356629783 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356629783 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3IcX | simonw 9599 | 2022-12-18T02:18:43Z | 2022-12-18T02:18:43Z | OWNER | Various attempts at a fix which didn't work: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b..899d36fd 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,4 +1,3 @@ -import asyncio import httpx import os import pathlib @@ -6,6 +5,7 @@ import pytest import pytest_asyncio import re import subprocess +import sys import tempfile import time import trustme @@ -27,13 +27,23 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None -def wait_until_responds(url, timeout=5.0, client=httpx, kwargs): +def wait_until_responds(url, timeout=5.0, client=None, kwargs): + client = client or httpx.Client(kwargs) start = time.time() while time.time() - start < timeout: try: - client.get(url, kwargs) + if "verify" in kwargs: + print(kwargs["verify"]) + print( + "Contents of verify file: {}".format( + open(kwargs.get("verify")).read() + ) + ) + print("client = {}, kwargs = {}".format(client, kwargs)) + client.get(url) return - except httpx.ConnectError: + except (httpx.ConnectError, httpx.RemoteProtocolError) as ex: + print(ex) time.sleep(0.1) raise AssertionError("Timed out waiting for {} to respond".format(url)) @@ -166,7 +176,7 @@ def check_permission_actions_are_documented(): @pytest.fixture(scope="session") def ds_localhost_http_server(): ds_proc = subprocess.Popen( - ["datasette", "--memory", "-p", "8041"], + [sys.executable, "-m", "datasette", "--memory", "-p", "8041"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, # Avoid FileNotFoundError: [Errno 2] No such file or directory: @@ -180,7 +190,7 @@ def ds_localhost_http_server(): ds_proc.terminate() -@pytest.fixture(scope="session") +@pytest.fixture def ds_localhost_https_server(tmp_path_factory): cert_directory = tmp_path_factory.mktemp("certs") ca = trustme.CA() @@ -194,6 +204,8 @@ def ds_localhost_https_server(tmp_path_factory): ca.cert_pem.write_to_path(path=client_cert) ds_proc = subprocess.Popen( [ + sys.executable, + "-m", "datasette", "--memory", "-p", @@ -207,7 +219,11 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - wait_until_responds("http://localhost:8042/", verify=client_cert) + wait_until_responds( + "http://localhost:8042/memory.json", + verify=client_cert, + headers={"Connection": "close"}, + ) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert diff --git a/tests/test_cli_serve_server.py b/tests/test_cli_serve_server.py index 1c31e2a3..9320b623 100644 --- a/tests/test_cli_serve_server.py +++ b/tests/test_cli_serve_server.py @@ -16,7 +16,11 @@ def test_serve_localhost_http(ds_localhost_http_server): @pytest.mark.serial def test_serve_localhost_https(ds_localhost_https_server): , client_cert = ds_localhost_https_server - response = httpx.get("https://localhost:8042/_memory.json", verify=client_cert) + response = httpx.get( + "https://localhost:8042/_memory.json", + verify=client_cert, + headers={"Connection": "close"}, + ) assert { "database": "_memory", "path": "/_memory", ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356627931 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356627931 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3H_b | simonw 9599 | 2022-12-18T02:13:01Z | 2022-12-18T02:13:01Z | OWNER | Rather than continue to bang my head against this, I'm tempted to rewrite this test to happen outside of Python world - in a bash script run by GitHub Actions, for example. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356627331 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356627331 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3H2D | simonw 9599 | 2022-12-18T02:11:17Z | 2022-12-18T02:11:17Z | OWNER | This issue might be relevant, but I tried the suggested fix in there ( |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356626334 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356626334 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Hme | simonw 9599 | 2022-12-18T02:04:01Z | 2022-12-18T02:04:07Z | OWNER | I used the steps to test manually from this comment: https://github.com/simonw/datasette/issues/1221#issuecomment-777901052 In one terminal:
So the feature still works, it's just the test that is broken for some reason. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356625642 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356625642 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Hbq | simonw 9599 | 2022-12-18T02:00:57Z | 2022-12-18T02:00:57Z | OWNER | I added the TLS support here: - https://github.com/simonw/datasette/issues/1221 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356625556 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356625556 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3HaU | simonw 9599 | 2022-12-18T02:00:18Z | 2022-12-18T02:00:18Z | OWNER | Maybe the reason the ASGI lifespan stuff broke was this line: https://github.com/simonw/datasette/blob/8b73fc6b47dffd8836f5c58aae1e57c1f66a5754/datasette/cli.py#L630-L632 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356620233 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356620233 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3GHJ | simonw 9599 | 2022-12-18T01:31:10Z | 2022-12-18T01:31:10Z | OWNER | During the polling loop it constantly raises:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356618913 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356618913 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Fyh | simonw 9599 | 2022-12-18T01:29:05Z | 2022-12-18T01:29:05Z | OWNER | Now the only failure is in the ``` message = str(exc)
/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/httpx/_transports/default.py:77: RemoteProtocolError =========================== short test summary info ============================ ERROR tests/test_cli_serve_server.py::test_serve_localhost_https - httpx.RemoteProtocolError: Server disconnected without sending a response. ================= 30 passed, 1264 deselected, 1 error in 6.15s ================= ``` That's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L16-L24 And this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L178-L215 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356610089 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356610089 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Dop | simonw 9599 | 2022-12-18T01:12:39Z | 2022-12-18T01:12:39Z | OWNER | ... and it turns out those tests saved me. Because I forgot to check if
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356609095 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356609095 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3DZH | simonw 9599 | 2022-12-18T01:10:43Z | 2022-12-18T01:10:43Z | OWNER | Improved version of that fixture: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 44c44f87..69dee68b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -27,6 +27,17 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None +def wait_until_responds(url, timeout=5.0, client=httpx, kwargs): + start = time.time() + while time.time() - start < timeout: + try: + client.get(url, kwargs) + return + except httpx.ConnectError: + time.sleep(0.1) + raise AssertionError("Timed out waiting for {} to respond".format(url)) + + @pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette @@ -161,13 +172,7 @@ def ds_localhost_http_server(): # Avoid FileNotFoundError: [Errno 2] No such file or directory: cwd=tempfile.gettempdir(), ) - # Loop until port 8041 serves traffic - while True: - try: - httpx.get("http://localhost:8041/") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8041/") # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc @@ -202,12 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - while True: - try: - httpx.get("https://localhost:8042/", verify=client_cert) - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8042/", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert @@ -231,12 +231,7 @@ def ds_unix_domain_socket_server(tmp_path_factory): # Poll until available transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport) - while True: - try: - client.get("http://localhost/_memory.json") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost/_memory.json", client=client) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, uds ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356600917 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356600917 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3BZV | simonw 9599 | 2022-12-18T01:02:26Z | 2022-12-18T01:02:26Z | OWNER | This bit here looks like it could hang!
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356599930 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356599930 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3BJ6 | simonw 9599 | 2022-12-18T01:01:47Z | 2022-12-18T01:01:47Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | ||
1356596740 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356596740 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3AYE | simonw 9599 | 2022-12-18T00:59:47Z | 2022-12-18T00:59:47Z | OWNER | Hitting ``` % pytest -m serial tests/test_cli_serve_server.py --full-trace ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 3 items tests/test_cli_serve_server.py ^C^C ====================================================== no tests ran in 3.49s ======================================================= Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions yield File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/backends/sync.py", line 86, in connect_tcp sock = socket.create_connection( File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 845, in create_connection raise err File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 61] Connection refused [...] ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356595665 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356595665 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3AHR | simonw 9599 | 2022-12-18T00:58:16Z | 2022-12-18T00:58:16Z | OWNER |
``` (datasette) datasette % pytest -m serial ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items / 1264 deselected / 31 selected tests/test_package.py . [ 3%] tests/test_cli_serve_server.py ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356489200 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356489200 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q2mHw | simonw 9599 | 2022-12-17T22:29:51Z | 2022-12-17T22:29:51Z | OWNER | No, it still causes the tests to hang (I let them run for 12 minutes): Interesting that the regular tests passed an then the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356487139 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356487139 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q2lnj | simonw 9599 | 2022-12-17T22:16:52Z | 2022-12-17T22:16:52Z | OWNER | I'm trying this fix again, after a bunch of work on the test suite in: - #1959 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356481595 | https://github.com/simonw/datasette/issues/1843#issuecomment-1356481595 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5Q2kQ7 | simonw 9599 | 2022-12-17T22:02:40Z | 2022-12-17T22:02:40Z | OWNER | This is interesting:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1356480256 | https://github.com/simonw/datasette/issues/1843#issuecomment-1356480256 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5Q2j8A | simonw 9599 | 2022-12-17T22:01:13Z | 2022-12-17T22:01:13Z | OWNER | The refactor in the following issue did NOT prevent this error from occurring when I try to run the full
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1356478792 | https://github.com/simonw/datasette/issues/1959#issuecomment-1356478792 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5Q2jlI | simonw 9599 | 2022-12-17T21:49:36Z | 2022-12-17T21:49:36Z | OWNER | Made a really good start on this in the just-merged PR: - #1960 The follow-up work will happen in: - #1962 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1356478575 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356478575 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q2jhv | simonw 9599 | 2022-12-17T21:47:48Z | 2022-12-17T21:47:48Z | OWNER | Stick a twig in it, this will do for the moment. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356476886 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356476886 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q2jHW | simonw 9599 | 2022-12-17T21:37:05Z | 2022-12-17T21:37:05Z | OWNER | I think this test may be to blame: It's over-riding |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356476583 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356476583 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q2jCn | simonw 9599 | 2022-12-17T21:34:51Z | 2022-12-17T21:34:51Z | OWNER | These are all the places that tests touch test_permissions.py 62: padlock_client.ds._metadata_local["allow"] = allow 77: del padlock_client.ds._metadata_local["allow"] 522: cascade_app_client.ds._metadata_local = updated_metadata 533: cascade_app_client.ds._metadata_local = previous_metadata 549: previous_metadata = cascade_app_client.ds._metadata_local 551: cascade_app_client.ds._metadata_local = metadata 566: cascade_app_client.ds._metadata_local = previous_metadata 842: perms_ds._metadata_local = updated_metadata 849: perms_ds._metadata_local = previous_metadata test_html.py 1114: orig = ds_client.ds._metadata_local 1115: ds_client.ds._metadata_local = metadata 1123: ds_client.ds._metadata_local = orig test_plugins.py 1034: ds_client.ds._metadata_local = { ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356056018 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356056018 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q08XS | simonw 9599 | 2022-12-17T05:33:26Z | 2022-12-17T05:33:26Z | OWNER | I think I've found the problem. The failing test is this one:
That's this example: Why is it expected to take 9 pages and not 5, when the definition of that view is this:
So this test is showing that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356038242 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356038242 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q04Bi | simonw 9599 | 2022-12-17T04:56:22Z | 2022-12-17T04:57:04Z | OWNER | May have spotted the problem with that
No that's not it either - |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355823260 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355823260 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q0Dic | simonw 9599 | 2022-12-16T23:36:07Z | 2022-12-16T23:36:07Z | OWNER | I ran ``` assert expected_rows == len(fetched)
On a hunch, I checked the current settings:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355691595 | https://github.com/simonw/datasette/issues/1962#issuecomment-1355691595 | https://api.github.com/repos/simonw/datasette/issues/1962 | IC_kwDOBm6k_c5QzjZL | simonw 9599 | 2022-12-16T21:53:45Z | 2022-12-16T21:55:29Z | OWNER | Also need an alternative mechanism to this convenience for getting CSRF tokens before a POST: One option would be adding that mechanism to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Alternative, async-friendly pattern for `make_app_client()` and similar - fully retire `TestClient` 1500636982 | |
1355685828 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355685828 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Qzh_E | simonw 9599 | 2022-12-16T21:50:01Z | 2022-12-16T21:50:01Z | OWNER | Looks like that
So I'm going to remove the mark. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355478743 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355478743 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyvbX | simonw 9599 | 2022-12-16T19:27:12Z | 2022-12-16T19:27:12Z | OWNER | Bad news: they're definitely caused by tests that are subtly affected by other tests. This passes without errors:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355475671 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355475671 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyurX | simonw 9599 | 2022-12-16T19:26:08Z | 2022-12-16T19:26:08Z | OWNER | Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355471341 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355471341 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Qytnt | simonw 9599 | 2022-12-16T19:23:33Z | 2022-12-16T19:23:33Z | OWNER | Trying this instead:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355445710 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355445710 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QynXO | simonw 9599 | 2022-12-16T19:06:49Z | 2022-12-16T19:09:35Z | OWNER | This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop. https://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though. https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/ |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355325426 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355325426 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyJ_y | simonw 9599 | 2022-12-16T18:00:40Z | 2022-12-16T18:00:40Z | OWNER | Many of the failing tests pass on my laptop but fail in CI. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355319541 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355319541 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyIj1 | simonw 9599 | 2022-12-16T17:58:24Z | 2022-12-16T17:58:46Z | OWNER |
It did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355317782 | https://github.com/simonw/datasette/issues/1959#issuecomment-1355317782 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QyIIW | simonw 9599 | 2022-12-16T17:57:25Z | 2022-12-16T17:57:25Z | OWNER | Opened a follow-up issue here: - #1962 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1355317369 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355317369 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyIB5 | simonw 9599 | 2022-12-16T17:57:14Z | 2022-12-16T17:57:14Z | OWNER | Opened a follow-up issue here: - #1962 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355313058 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355313058 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyG-i | simonw 9599 | 2022-12-16T17:53:17Z | 2022-12-16T17:53:17Z | OWNER | Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626 ```
I tried adding |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355300217 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355300217 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyD15 | simonw 9599 | 2022-12-16T17:44:55Z | 2022-12-16T17:44:55Z | OWNER | That's enough for this round. I'll get the tests passing and land this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354269873 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354269873 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QuISx | simonw 9599 | 2022-12-16T06:11:43Z | 2022-12-16T06:11:43Z | OWNER | This is quite fast:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354148139 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354148139 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Qtqkr | simonw 9599 | 2022-12-16T03:32:52Z | 2022-12-16T03:32:52Z | OWNER | Got that done to: 68 passed in 14.92s By implementing my own global variable - since |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354072344 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354072344 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtYEY | simonw 9599 | 2022-12-16T02:00:07Z | 2022-12-16T02:00:07Z | OWNER | It did NOT speed it up: 68 passed in 26.26s |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354061440 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354061440 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtVaA | simonw 9599 | 2022-12-16T01:45:38Z | 2022-12-16T01:45:38Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | ||
1354053151 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354053151 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtTYf | simonw 9599 | 2022-12-16T01:33:22Z | 2022-12-16T01:33:22Z | OWNER | The thing with The problem is that the tests written against But the new This results in a datasette instance that DOES have a My initial solution attempt was a huge hack - I decided that if you pass This is weird and surprising (
So I'm going to undo that hack and teach the fixture to do this instead:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354046627 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354046627 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtRyj | simonw 9599 | 2022-12-16T01:23:21Z | 2022-12-16T01:23:21Z | OWNER | This does seem to help: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 1306c407..af9c7696 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = { } -@pytest.fixture(scope="session") -def event_loop(): - return asyncio.get_event_loop() - - -@pytest_asyncio.fixture(scope="session") +@pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette from .fixtures import METADATA, PLUGINS_DIR @@ -53,10 +48,11 @@ async def ds_client(): db = ds.add_memory_database("fixtures")
Need to get |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354036967 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354036967 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtPbn | simonw 9599 | 2022-12-16T01:10:12Z | 2022-12-16T01:10:12Z | OWNER | If it does turn out that I can't use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354023960 | https://github.com/simonw/datasette/issues/1958#issuecomment-1354023960 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtMQY | simonw 9599 | 2022-12-16T00:58:14Z | 2022-12-16T00:58:19Z | OWNER | This is in the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1354019543 | https://github.com/simonw/datasette/issues/1958#issuecomment-1354019543 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtLLX | simonw 9599 | 2022-12-16T00:54:11Z | 2022-12-16T00:54:11Z | OWNER | To test the fix, I did a fresh checkout of RUN apt-get update && \ apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg/info/* RUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \ find /usr/local/lib -name 'pycache' | xargs rm -r && \ rm -rf /root/.cache/pip EXPOSE 8001
CMD ["datasette"]
Then I pushed a changed to branch and changed the line to:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1354008688 | https://github.com/simonw/datasette/issues/1958#issuecomment-1354008688 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtIhw | simonw 9599 | 2022-12-16T00:45:40Z | 2022-12-16T00:45:40Z | OWNER | The fix may just be to switch to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1353977605 | https://github.com/simonw/datasette/issues/1958#issuecomment-1353977605 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtA8F | simonw 9599 | 2022-12-16T00:38:23Z | 2022-12-16T00:38:23Z | OWNER | Confirmed, I just got the same result: ``` - % docker run datasetteproject/datasette pip install datasette-upload-csvs ~ % docker commit $(docker ps -lq) datasette-with-plugins sha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f ~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0 INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ^CINFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [1]
http://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1353812913 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353812913 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsYux | simonw 9599 | 2022-12-15T22:48:54Z | 2022-12-15T22:48:54Z | OWNER | This is all very broken: ``` % pytest -x --pdb ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items tests/test_package.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_cli_serve_get.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_black.py . [ 0%] tests/test_api.py E
fixturedef = <FixtureDef argname='event_loop' scope='session' baseid='tests'>, request = <SubRequest 'event_loop' for \<Function test_homepage>>
/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pytest_asyncio/plugin.py:377: /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: in event_loop return asyncio.get_event_loop() self = <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x111e34670>
/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py:656: RuntimeError
=================================================================================== warnings summary ==================================================================================== tests/test_cli.py::test_inspect_cli_writes_to_file tests/test_cli.py::test_inspect_cli /Users/simon/Dropbox/Development/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop loop = asyncio.get_event_loop() tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error tests/test_cli_serve_get.py::test_serve_with_get /Users/simon/Dropbox/Development/datasette/datasette/cli.py:596: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(ds.invoke_startup()) tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error tests/test_cli_serve_get.py::test_serve_with_get /Users/simon/Dropbox/Development/datasette/datasette/cli.py:599: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(check_databases(ds)) tests/test_api.py::test_homepage /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: DeprecationWarning: There is no current event loop return asyncio.get_event_loop() -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================================ short test summary info ================================================================================ ERROR tests/test_api.py::test_homepage - RuntimeError: There is no current event loop in thread 'MainThread'. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Quitting debugger !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ======================================================================== 7 passed, 7 warnings, 1 error in 19.15s ======================================================================== (datasette) datasette % ``` The problem looks to be caused by this: Which I found necessary in order to have Can I work around this, or is |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353805839 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353805839 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsXAP | simonw 9599 | 2022-12-15T22:38:37Z | 2022-12-15T22:38:37Z | OWNER | I'm going to make |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353765125 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353765125 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsNEF | simonw 9599 | 2022-12-15T22:00:04Z | 2022-12-15T22:00:04Z | OWNER | I'm going to punt on that for the moment and continue to use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353763837 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353763837 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsMv9 | simonw 9599 | 2022-12-15T21:59:05Z | 2022-12-15T21:59:05Z | OWNER | Here's an annoying error: ```
tests/test_auth.py:88: self = <datasette.app.DatasetteClient object at 0x1033cc4f0>, path = '/-/logout' kwargs = {'cookies': {'ds_actor': 'eyJhIjp7ImlkIjoidGVzdCJ9fQ.fuFCTJG5XE-RNnUM7dcnXx9sPvE'}, 'csrftoken_from': True}, client = <httpx.AsyncClient object at 0x10347a9b0>
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353749401 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353749401 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsJOZ | simonw 9599 | 2022-12-15T21:47:27Z | 2022-12-15T21:47:27Z | OWNER | I'm using this new mark:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353747370 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353747370 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsIuq | simonw 9599 | 2022-12-15T21:45:14Z | 2022-12-15T21:45:14Z | OWNER | I'm going to do this in a PR. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353738075 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353738075 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsGdb | simonw 9599 | 2022-12-15T21:35:56Z | 2022-12-15T21:35:56Z | OWNER | I built that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353728682 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353728682 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsEKq | simonw 9599 | 2022-12-15T21:28:35Z | 2022-12-15T21:28:35Z | OWNER | Got this error trying to have two tests use the same Adding this to
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353721091 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsCUD | simonw 9599 | 2022-12-15T21:20:32Z | 2022-12-15T21:20:32Z | OWNER | Rather than tediously rewriting every single test to the new shape I'm going to try a wrapper for that HTTPX response that transforms it into an imitation of the one returned by the existing |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353720559 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353720559 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsCLv | simonw 9599 | 2022-12-15T21:19:56Z | 2022-12-15T21:19:56Z | OWNER | Here's a port of the first ```diff diff --git a/datasette/app.py b/datasette/app.py index f3cb8876..b770b469 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -281,7 +281,7 @@ class Datasette: raise self.crossdb = crossdb self.nolock = nolock - if memory or crossdb or not self.files: + if memory or crossdb or (not self.files and memory is not False): self.add_database( Database(self, is_mutable=False, is_memory=True), name="_memory" ) diff --git a/pytest.ini b/pytest.ini index 559e518c..0bcb0d1e 100644 --- a/pytest.ini +++ b/pytest.ini @@ -8,4 +8,5 @@ filterwarnings= ignore:.current_task.:PendingDeprecationWarning markers = serial: tests to avoid using with pytest-xdist + ds_client: tests using the ds_client fixture asyncio_mode = strict diff --git a/tests/conftest.py b/tests/conftest.py index cd735e12..648423ba 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -2,6 +2,7 @@ import httpx import os import pathlib import pytest +import pytest_asyncio import re import subprocess import tempfile @@ -23,6 +24,22 @@ UNDOCUMENTED_PERMISSIONS = { } +@pytest_asyncio.fixture +async def ds_client(): + from datasette.app import Datasette + from .fixtures import METADATA, PLUGINS_DIR + ds = Datasette(memory=False, metadata=METADATA, plugins_dir=PLUGINS_DIR) + from .fixtures import TABLES, TABLE_PARAMETERIZED_SQL + db = ds.add_memory_database("fixtures") + def prepare(conn): + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) + await db.execute_write_fn(prepare) + return ds.client + + def pytest_report_header(config): return "SQLite: {}".format( sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0] diff --git a/tests/test_api.py b/tests/test_api.py index 5f2a6ea6..ddf4219c 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -23,12 +23,15 @@ import sys import urllib -def test_homepage(app_client): - response = app_client.get("/.json") - assert response.status == 200 +@pytest.mark.ds_client +@pytest.mark.asyncio +async def test_homepage(ds_client): + response = await ds_client.get("/.json") + assert response.status_code == 200 assert "application/json; charset=utf-8" == response.headers["content-type"] - assert response.json.keys() == {"fixtures": 0}.keys() - d = response.json["fixtures"] + data = response.json() + assert data.keys() == {"fixtures": 0}.keys() + d = data["fixtures"] assert d["name"] == "fixtures" assert d["tables_count"] == 24 assert len(d["tables_and_views_truncated"]) == 5 @@ -36,7 +39,7 @@ def test_homepage(app_client): # 4 hidden FTS tables + no_primary_key (hidden in metadata) assert d["hidden_tables_count"] == 6 # 201 in no_primary_key, plus 6 in other hidden tables: - assert d["hidden_table_rows_sum"] == 207, response.json + assert d["hidden_table_rows_sum"] == 207, data assert d["views_count"] == 4 ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353707828 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353707828 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5Qr_E0 | simonw 9599 | 2022-12-15T21:06:29Z | 2022-12-15T21:06:29Z | OWNER | Previous, abandoned attempt at this work (for #1843): ```diff diff --git a/datasette/app.py b/datasette/app.py index 7e682498..cf35c3a2 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -228,7 +228,7 @@ class Datasette: template_dir=None, plugins_dir=None, static_mounts=None, - memory=False, + memory=None, settings=None, secret=None, version_note=None, @@ -238,6 +238,7 @@ class Datasette: nolock=False, ): self._startup_invoked = False + self._extra_on_startup = [] assert config_dir is None or isinstance( config_dir, Path ), "config_dir= should be a pathlib.Path" @@ -278,7 +279,7 @@ class Datasette: raise self.crossdb = crossdb self.nolock = nolock - if memory or crossdb or not self.files: + if memory or crossdb or (not self.files and memory is not False): self.add_database( Database(self, is_mutable=False, is_memory=True), name="_memory" ) @@ -391,6 +392,9 @@ class Datasette: self._root_token = secrets.token_hex(32) self.client = DatasetteClient(self)
diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py index 56690251..986755cb 100644 --- a/datasette/utils/asgi.py +++ b/datasette/utils/asgi.py @@ -423,9 +423,9 @@ class AsgiFileDownload: class AsgiRunOnFirstRequest: - def init(self, asgi, on_startup): + def init(self, app, on_startup): assert isinstance(on_startup, list) - self.asgi = asgi + self.app = app self.on_startup = on_startup self._started = False @@ -434,4 +434,4 @@ class AsgiRunOnFirstRequest: self._started = True for hook in self.on_startup: await hook() - return await self.asgi(scope, receive, send) + return await self.app(scope, receive, send) diff --git a/tests/conftest.py b/tests/conftest.py index cd735e12..d1301943 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -23,6 +23,15 @@ UNDOCUMENTED_PERMISSIONS = { } +# @pytest.fixture(autouse=True) +# def log_name_of_test_before_test(request): +# # To help identify tests that are hanging +# name = str(request.node) +# with open("/tmp/test.log", "a") as f: +# f.write(name + "\n") +# yield + + def pytest_report_header(config): return "SQLite: {}".format( sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0] diff --git a/tests/fixtures.py b/tests/fixtures.py index a6700239..18d3f1b7 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -101,6 +101,19 @@ EXPECTED_PLUGINS = [ ] +def _populate_connection(conn): + # Drop any tables and views that exist + to_drop = conn.execute( + "SELECT name, type FROM sqlite_master where type in ('table', 'view')" + ).fetchall() + for name, type in to_drop: + conn.execute(f"DROP {type} IF EXISTS [{name}]") + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) + + @contextlib.contextmanager def make_app_client( sql_time_limit_ms=None, @@ -117,45 +130,22 @@ def make_app_client( metadata=None, crossdb=False, ): - with tempfile.TemporaryDirectory() as tmpdir: - filepath = os.path.join(tmpdir, filename) - if is_immutable: - files = [] - immutables = [filepath] - else: - files = [filepath] - immutables = [] - conn = sqlite3.connect(filepath) - conn.executescript(TABLES) - for sql, params in TABLE_PARAMETERIZED_SQL: - with conn: - conn.execute(sql, params) - # Close the connection to avoid "too many open files" errors - conn.close() - if extra_databases is not None: - for extra_filename, extra_sql in extra_databases.items(): - extra_filepath = os.path.join(tmpdir, extra_filename) - c2 = sqlite3.connect(extra_filepath) - c2.executescript(extra_sql) - c2.close() - # Insert at start to help test /-/databases ordering: - files.insert(0, extra_filepath) - os.chdir(os.path.dirname(filepath)) - settings = settings or {} - for key, value in { - "default_page_size": 50, - "max_returned_rows": max_returned_rows or 100, - "sql_time_limit_ms": sql_time_limit_ms or 200, - # Default is 3 but this results in "too many open files" - # errors when running the full test suite: - "num_sql_threads": 1, - }.items(): - if key not in settings: - settings[key] = value + settings = settings or {} + for key, value in { + "default_page_size": 50, + "max_returned_rows": max_returned_rows or 100, + "sql_time_limit_ms": sql_time_limit_ms or 200, + # Default is 3 but this results in "too many open files" + # errors when running the full test suite: + "num_sql_threads": 1, + }.items(): + if key not in settings: + settings[key] = value + # We can use an in-memory database, but only if we're not doing anything + # with is_immutable or extra_databases and filename is the default + if not is_immutable and not extra_databases and filename == "fixtures.db": ds = Datasette( - files, - immutables=immutables, - memory=memory, + memory=memory or False, cors=cors, metadata=metadata or METADATA, plugins_dir=PLUGINS_DIR, @@ -165,12 +155,57 @@ def make_app_client( template_dir=template_dir, crossdb=crossdb, ) + db = ds.add_memory_database("fixtures") + + async def populate_fixtures(): + print("Here we go... populating fixtures") + await db.execute_write_fn(_populate_connection) + + ds._add_on_startup(populate_fixtures) yield TestClient(ds) - # Close as many database connections as possible - # to try and avoid too many open files error - for db in ds.databases.values(): - if not db.is_memory: - db.close() + else: + with tempfile.TemporaryDirectory() as tmpdir: + filepath = os.path.join(tmpdir, filename) + if is_immutable: + files = [] + immutables = [filepath] + else: + files = [filepath] + immutables = [] + + conn = sqlite3.connect(filepath) + _populate_connection(conn) + # Close the connection to reduce "too many open files" errors + conn.close() + + if extra_databases is not None: + for extra_filename, extra_sql in extra_databases.items(): + extra_filepath = os.path.join(tmpdir, extra_filename) + c2 = sqlite3.connect(extra_filepath) + c2.executescript(extra_sql) + c2.close() + # Insert at start to help test /-/databases ordering: + files.insert(0, extra_filepath) + os.chdir(os.path.dirname(filepath)) + ds = Datasette( + files, + immutables=immutables, + memory=memory, + cors=cors, + metadata=metadata or METADATA, + plugins_dir=PLUGINS_DIR, + settings=settings, + inspect_data=inspect_data, + static_mounts=static_mounts, + template_dir=template_dir, + crossdb=crossdb, + ) + yield TestClient(ds) + # Close as many database connections as possible + # to try and avoid too many open files error + for db in ds.databases.values(): + if not db.is_memory: + db.close() @pytest.fixture(scope="session") diff --git a/tests/test_cli.py b/tests/test_cli.py index d3e015fa..d9e4e457 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,5 +1,6 @@ from .fixtures import ( app_client, + app_client_with_cors, make_app_client, TestClient as _TestClient, EXPECTED_PLUGINS, @@ -38,7 +39,7 @@ def test_inspect_cli(app_client): assert expected_count == database["tables"][table_name]["count"] -def test_inspect_cli_writes_to_file(app_client): +def test_inspect_cli_writes_to_file(app_client_with_cors): runner = CliRunner() result = runner.invoke( cli, ["inspect", "fixtures.db", "--inspect-file", "foo.json"] ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353705072 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353705072 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5Qr-Zw | simonw 9599 | 2022-12-15T21:04:07Z | 2022-12-15T21:04:07Z | OWNER | I'm going to start by getting every test that uses the raw |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
issue >30