html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/issues/1955#issuecomment-1356487139,https://api.github.com/repos/simonw/datasette/issues/1955,1356487139,IC_kwDOBm6k_c5Q2lnj,9599,2022-12-17T22:16:52Z,2022-12-17T22:16:52Z,OWNER,"I'm trying this fix again, after a bunch of work on the test suite in: - #1959","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1843#issuecomment-1356481595,https://api.github.com/repos/simonw/datasette/issues/1843,1356481595,IC_kwDOBm6k_c5Q2kQ7,9599,2022-12-17T22:02:40Z,2022-12-17T22:02:40Z,OWNER,"This is interesting: ```python import resource print(resource.getrlimit(resource.RLIMIT_NOFILE)) ``` On Mac: ```pycon >>> print(resource.getrlimit(resource.RLIMIT_NOFILE)) (256, 9223372036854775807) ``` On Ubuntu (in a Docker `ubuntu:22.04` container): ```pycon >>> resource.getrlimit(resource.RLIMIT_NOFILE) (1048576, 1048576) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1356480256,https://api.github.com/repos/simonw/datasette/issues/1843,1356480256,IC_kwDOBm6k_c5Q2j8A,9599,2022-12-17T22:01:13Z,2022-12-17T22:01:13Z,OWNER,"The refactor in the following issue did NOT prevent this error from occurring when I try to run the full `pytest` suite on my Mac laptop: - #1959","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1959#issuecomment-1356478792,https://api.github.com/repos/simonw/datasette/issues/1959,1356478792,IC_kwDOBm6k_c5Q2jlI,9599,2022-12-17T21:49:36Z,2022-12-17T21:49:36Z,OWNER,"Made a really good start on this in the just-merged PR: - #1960 The follow-up work will happen in: - #1962","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664, https://github.com/simonw/datasette/pull/1960#issuecomment-1356478575,https://api.github.com/repos/simonw/datasette/issues/1960,1356478575,IC_kwDOBm6k_c5Q2jhv,9599,2022-12-17T21:47:48Z,2022-12-17T21:47:48Z,OWNER,"Stick a twig in it, this will do for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1354062939,https://api.github.com/repos/simonw/datasette/issues/1960,1354062939,IC_kwDOBm6k_c5QtVxb,22429695,2022-12-16T01:48:06Z,2022-12-17T21:40:43Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.19**% // Head: **92.05**% // Decreases project coverage by **`-0.13%`** :warning: > Coverage data is based on head [(`770879a`)](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`0b68996`)](https://codecov.io/gh/simonw/datasette/commit/0b68996cc511b3a801f0cd0157bd66332d75f46f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 100.00% of modified lines in pull request are covered. > :exclamation: Current head 770879a differs from pull request most recent head f42bca8. Consider uploading reports for the commit f42bca8 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1960 +/- ## ========================================== - Coverage 92.19% 92.05% -0.14% ========================================== Files 38 38 Lines 5521 5527 +6 ========================================== - Hits 5090 5088 -2 - Misses 431 439 +8 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.13% <100.00%> (-0.57%)` | :arrow_down: | | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.83% <100.00%> (+0.24%)` | :arrow_up: | | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: | | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.57% <0.00%> (-0.61%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1356476886,https://api.github.com/repos/simonw/datasette/issues/1960,1356476886,IC_kwDOBm6k_c5Q2jHW,9599,2022-12-17T21:37:05Z,2022-12-17T21:37:05Z,OWNER,"I think this test may be to blame: https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/tests/test_plugins.py#L950-L972 It's over-riding `_metadata_local` and then failing to set it back to original in a `finally:` block at the end.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1356476583,https://api.github.com/repos/simonw/datasette/issues/1960,1356476583,IC_kwDOBm6k_c5Q2jCn,9599,2022-12-17T21:34:51Z,2022-12-17T21:34:51Z,OWNER,"These are all the places that tests touch `_metadata_local` at the moment: ``` (venv) root@76a81d2417f5:/tmp/datasette/tests# rg _metadata_local test_facets.py 596: ds._metadata_local = { 605: ds._metadata_local[""databases""][""test_facet_size""][""tables""][""neighbourhoods""][ test_permissions.py 62: padlock_client.ds._metadata_local[""allow""] = allow 77: del padlock_client.ds._metadata_local[""allow""] 522: cascade_app_client.ds._metadata_local = updated_metadata 533: cascade_app_client.ds._metadata_local = previous_metadata 549: previous_metadata = cascade_app_client.ds._metadata_local 551: cascade_app_client.ds._metadata_local = metadata 566: cascade_app_client.ds._metadata_local = previous_metadata 842: perms_ds._metadata_local = updated_metadata 849: perms_ds._metadata_local = previous_metadata test_html.py 1114: orig = ds_client.ds._metadata_local 1115: ds_client.ds._metadata_local = metadata 1123: ds_client.ds._metadata_local = orig test_plugins.py 1034: ds_client.ds._metadata_local = { ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1356056018,https://api.github.com/repos/simonw/datasette/issues/1960,1356056018,IC_kwDOBm6k_c5Q08XS,9599,2022-12-17T05:33:26Z,2022-12-17T05:33:26Z,OWNER,"I think I've found the problem. The failing test is this one: `paginated_view.json-201-9` That's this example: https://github.com/simonw/datasette/blob/cede1efeedbc3d928397d53d5a1611eecc598fde/tests/test_table_api.py#L179-L180 Why is it expected to take 9 pages and not 5, when the definition of that view is this: ```sql CREATE VIEW paginated_view AS SELECT content, '- ' || content || ' -' AS content_extra FROM no_primary_key; ``` Because `paginated_view` has extra configuration in `metadata.json`: https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/tests/fixtures.py#L357 So this test is showing that `metadata` can be used to set an alternative page size for a view. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1356038242,https://api.github.com/repos/simonw/datasette/issues/1960,1356038242,IC_kwDOBm6k_c5Q04Bi,9599,2022-12-17T04:56:22Z,2022-12-17T04:57:04Z,OWNER,"May have spotted the problem with that `test_paginate_tables_and_views` test: ``` (Pdb) path, expected_rows, expected_pages (None, 201, 9) ``` `path` should not be `None` here: https://github.com/simonw/datasette/blob/cede1efeedbc3d928397d53d5a1611eecc598fde/tests/test_table_api.py#L175-L212 No that's not it either - `path` is reassigned on purpose.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1355823260,https://api.github.com/repos/simonw/datasette/issues/1960,1355823260,IC_kwDOBm6k_c5Q0Dic,9599,2022-12-16T23:36:07Z,2022-12-16T23:36:07Z,OWNER,"I ran `pytest --pdb -x` to drop into the debugger on the first failing test. ``` assert expected_rows == len(fetched) > assert expected_pages == count E assert 9 == 5 ``` That's in `test_paginate_tables_and_views()`. On a hunch, I checked the current settings: ``` (Pdb) ds_client.ds._settings {'default_page_size': 50, 'max_returned_rows': 100, 'max_insert_rows': 100, 'num_sql_threads': 1, 'sql_time_limit_ms': 200, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'allow_facet': True, 'allow_download': True, 'allow_signed_tokens': True, 'max_signed_tokens_ttl': 0, 'suggest_facets': True, 'default_cache_ttl': 5, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'trace_debug': False, 'base_url': '/'} ``` Looks like something changed `default_page_size` to 50 and forgot to change it back!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/issues/1962#issuecomment-1355691595,https://api.github.com/repos/simonw/datasette/issues/1962,1355691595,IC_kwDOBm6k_c5QzjZL,9599,2022-12-16T21:53:45Z,2022-12-16T21:55:29Z,OWNER,"Also need an alternative mechanism to this convenience for getting CSRF tokens before a POST: https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/datasette/utils/testing.py#L90-L103 One option would be adding that mechanism to `datasette.client.post(...)` - as a `_csrf_from=` parameter (with an underscore because it's mainly intended for use in tests, though perhaps that's a weird convention that I shouldn't introduce).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1500636982, https://github.com/simonw/datasette/pull/1960#issuecomment-1355685828,https://api.github.com/repos/simonw/datasette/issues/1960,1355685828,IC_kwDOBm6k_c5Qzh_E,9599,2022-12-16T21:50:01Z,2022-12-16T21:50:01Z,OWNER,"Looks like that `@pytest.mark.ds_client` mark I've been using isn't necessary - I added that so I could easily run `pytest -m ds_client` to execute all tests that I had ported to the new feature, but actually this achieves the same thing: pytest -k ds_client So I'm going to remove the mark.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1355478743,https://api.github.com/repos/simonw/datasette/issues/1960,1355478743,IC_kwDOBm6k_c5QyvbX,9599,2022-12-16T19:27:12Z,2022-12-16T19:27:12Z,OWNER,"Bad news: they're definitely caused by tests that are subtly affected by other tests. This passes without errors: pytest -k test_paginate_tables_and_views","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1355475671,https://api.github.com/repos/simonw/datasette/issues/1960,1355475671,IC_kwDOBm6k_c5QyurX,9599,2022-12-16T19:26:08Z,2022-12-16T19:26:08Z,OWNER,"Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI: ``` ======== short test summary info ======== FAILED tests/test_table_api.py::test_paginate_tables_and_views[/fixtures/paginated_view.json-201-9] - assert 9 == 5 FAILED tests/test_table_api.py::test_sortable_columns_metadata - KeyError: 'error' FAILED tests/test_table_api.py::test_searchable_views[/fixtures/searchable_view_configured_by_metadata.json?_search=weasel-expected_rows0] - AssertionError: assert [[2, 'terry d...sel', 'puma']] == [[1, 'barry c...sel', 'puma']] FAILED tests/test_table_api.py::test_unit_filters - KeyError: 'distance' FAILED tests/test_table_api.py::test_custom_query_with_unicode_characters - AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...] FAILED tests/test_table_html.py::test_table_html_no_primary_key - AttributeError: 'NoneType' object has no attribute 'strip' FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['']] == [['1']] FAILED tests/test_table_html.py::test_view_html - assert upper_content is None FAILED tests/test_table_html.py::test_table_metadata - AssertionError: assert 'This <em&...t; is escaped' == 'simple_primary_key' FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id\xa0▼', 'name'] FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk\xa0▼', 'name'] FAILED tests/test_table_html.py::test_column_metadata - AttributeError: 'NoneType' object has no attribute 'findAll' ======== 13 failed, 1279 passed, 3 skipped, 57 warnings in 572.40s (0:09:32) ======== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1355471341,https://api.github.com/repos/simonw/datasette/issues/1960,1355471341,IC_kwDOBm6k_c5Qytnt,9599,2022-12-16T19:23:33Z,2022-12-16T19:23:33Z,OWNER,"Trying this instead: ``` docker run -it ubuntu:22.04 /bin/bash ``` Then in that shell: ``` apt-get update apt-get install python3.11 python3.11-venv git -y cd /tmp git clone https://github.com/simonw/datasette cd datasette git checkout async-tests python3.11 -m venv venv pip install -e '.[test]' pytest ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1355445710,https://api.github.com/repos/simonw/datasette/issues/1960,1355445710,IC_kwDOBm6k_c5QynXO,9599,2022-12-16T19:06:49Z,2022-12-16T19:09:35Z,OWNER,"This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop. https://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though. https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1355325426,https://api.github.com/repos/simonw/datasette/issues/1960,1355325426,IC_kwDOBm6k_c5QyJ_y,9599,2022-12-16T18:00:40Z,2022-12-16T18:00:40Z,OWNER,Many of the failing tests pass on my laptop but fail in CI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1355319541,https://api.github.com/repos/simonw/datasette/issues/1960,1355319541,IC_kwDOBm6k_c5QyIj1,9599,2022-12-16T17:58:24Z,2022-12-16T17:58:46Z,OWNER,"> I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this. It did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884 ====== 11 failed, 1252 passed, 1 skipped, 1 warning in 185.77s (0:03:05) =======","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/issues/1959#issuecomment-1355317782,https://api.github.com/repos/simonw/datasette/issues/1959,1355317782,IC_kwDOBm6k_c5QyIIW,9599,2022-12-16T17:57:25Z,2022-12-16T17:57:25Z,OWNER,"Opened a follow-up issue here: - #1962","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664, https://github.com/simonw/datasette/pull/1960#issuecomment-1355317369,https://api.github.com/repos/simonw/datasette/issues/1960,1355317369,IC_kwDOBm6k_c5QyIB5,9599,2022-12-16T17:57:14Z,2022-12-16T17:57:14Z,OWNER,"Opened a follow-up issue here: - #1962","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1355313058,https://api.github.com/repos/simonw/datasette/issues/1960,1355313058,IC_kwDOBm6k_c5QyG-i,9599,2022-12-16T17:53:17Z,2022-12-16T17:53:17Z,OWNER,"Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626 ``` > assert response.json() == [{""id"": 1, ""name"": ""San Francisco""}] E AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] E Full diff: E - [{'id': 1, 'name': 'San Francisco'}] E + {'error': 'Table not found: 𝐜𝐢𝐭𝐢𝐞𝐬', 'ok': False, 'status': 404, 'title': None} ``` A hunch: this failure suggests that maybe the fixtures tables were not correctly created when this test run. Maybe that can happen when `python -n auto` runs a bunch of separate processes and hence one of the tests randomly gets run in a fresh process and executes before the in-memory fixtures database has been fully populated. I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1355300217,https://api.github.com/repos/simonw/datasette/issues/1960,1355300217,IC_kwDOBm6k_c5QyD15,9599,2022-12-16T17:44:55Z,2022-12-16T17:44:55Z,OWNER,That's enough for this round. I'll get the tests passing and land this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1354269873,https://api.github.com/repos/simonw/datasette/issues/1960,1354269873,IC_kwDOBm6k_c5QuISx,9599,2022-12-16T06:11:43Z,2022-12-16T06:11:43Z,OWNER,"This is quite fast: ``` % pytest -m ds_client -n auto ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict gw0 [291] / gw1 [291] / gw2 [291] / gw3 [291] / gw4 [291] / gw5 [291] ................................................................................................................................................................................. [ 60%] .................................................................................................................. [100%] ================================================================================== 291 passed in 6.30s ================================================================================== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354192168,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1354192168,IC_kwDOCGYnMM5Qt1Uo,9020979,2022-12-16T04:35:30Z,2022-12-16T04:35:38Z,NONE,"A related historical problem: https://github.com/tekartik/sqflite/issues/525#issuecomment-714500720 I wonder if the version of Sqlite or Python for Intel chip have defensive mode disabled by default, whereas M1 chips versions have it enabled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354160286,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1354160286,IC_kwDOCGYnMM5Qttie,9020979,2022-12-16T03:51:19Z,2022-12-16T03:52:13Z,NONE,"Hi @ryascott, thanks for sharing this! How did you upgrade your sqlite3 version? I'm running into this issue (also on an m1) with `Python ==3.10.7 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30` Unfortunately, `3.10.8` isn't listed in `pyenv` so I'm unable to install it. For me, the trigger is trying to use the `add-foreign-key` command on its own: ```bash sqlite-utils add-foreign-key library.db book_creators creator_id creators id ``` Some stackoverflow searching suggests that brew installing sqlite may fix it ( https://stackoverflow.com/questions/26345972/how-do-i-upgrade-the-sqlite-version-used-by-pythons-sqlite3-module-on-mac ), but I don't want to risk breaking the version of sqlite used by some other system, I'd only like to upgrade sqlite3 inside my current virtual environment. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495, https://github.com/simonw/datasette/pull/1960#issuecomment-1354148139,https://api.github.com/repos/simonw/datasette/issues/1960,1354148139,IC_kwDOBm6k_c5Qtqkr,9599,2022-12-16T03:32:52Z,2022-12-16T03:32:52Z,OWNER,"Got that done to: 68 passed in 14.92s By implementing my own global variable - since `pytest` won't use a global fixture for me, I decided to do it for myself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1354072344,https://api.github.com/repos/simonw/datasette/issues/1960,1354072344,IC_kwDOBm6k_c5QtYEY,9599,2022-12-16T02:00:07Z,2022-12-16T02:00:07Z,OWNER,"It did NOT speed it up: 68 passed in 26.26s","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1354061440,https://api.github.com/repos/simonw/datasette/issues/1960,1354061440,IC_kwDOBm6k_c5QtVaA,9599,2022-12-16T01:45:38Z,2022-12-16T01:45:38Z,OWNER,"I'm going to do `test_table_html.py` next. Currently: 68 passed in 17.20s Will this speed it up?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1354053151,https://api.github.com/repos/simonw/datasette/issues/1960,1354053151,IC_kwDOBm6k_c5QtTYf,9599,2022-12-16T01:33:22Z,2022-12-16T01:33:22Z,OWNER,"The thing with `Datasette(memory=False)` is tripping me up. The problem is that the tests written against `app_client` - which I want to replace - all assume that there is no `_memory` database, because when you start Datasette with at least one database file it doesn't enable `_memory` unless you explicitly tell it to. But the new `ds_client` fixture works by creating a named in-memory database called `fixtures`, which it does with a call to `ds.add_memory_database(""fixtures"")` after the object has been instantiated. This results in a datasette instance that DOES have a `_memory` database, when we didn't want one. My initial solution attempt was a huge hack - I decided that if you pass `memory=False` to the `Datasette` constructor it should mean ""don't add a `_memory` database even though I didn't pass any files"". I set a the default `memory` argument to `None`. This is weird and surprising (`memory=False` no does something different from `memory=None`?) and I found other tests that it broke, like this one: ```python def test_sql_errors_logged_to_stderr(): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, [""--get"", ""/_memory.json?sql=select+blah""]) assert result.exit_code == 1 assert ""sql = 'select blah', params = {}: no such column: blah\n"" in result.stderr ``` It ended up with no `_memory` database because it turns out `datasette serve ...` passes `memory=False` without me realizing it. So I'm going to undo that hack and teach the fixture to do this instead: ```python db = ds.add_memory_database(""fixtures"") ds.remove_database(""_memory"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1354046627,https://api.github.com/repos/simonw/datasette/issues/1960,1354046627,IC_kwDOBm6k_c5QtRyj,9599,2022-12-16T01:23:21Z,2022-12-16T01:23:21Z,OWNER,"This does seem to help: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 1306c407..af9c7696 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = { } -@pytest.fixture(scope=""session"") -def event_loop(): - return asyncio.get_event_loop() - - -@pytest_asyncio.fixture(scope=""session"") +@pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette from .fixtures import METADATA, PLUGINS_DIR @@ -53,10 +48,11 @@ async def ds_client(): db = ds.add_memory_database(""fixtures"") def prepare(conn): - conn.executescript(TABLES) - for sql, params in TABLE_PARAMETERIZED_SQL: - with conn: - conn.execute(sql, params) + if not conn.execute(""select count(*) from sqlite_master"").fetchone()[0]: + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) await db.execute_write_fn(prepare) return ds.client diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py index 4f7bf08c..d588342c 100644 --- a/tests/plugins/my_plugin_2.py +++ b/tests/plugins/my_plugin_2.py @@ -117,7 +117,12 @@ def actor_from_request(datasette, request): def permission_allowed(datasette, actor, action): # Testing asyncio version of permission_allowed async def inner(): - assert 2 == (await datasette.get_database().execute(""select 1 + 1"")).first()[0] + assert ( + 2 + == ( + await datasette.get_database(""_internal"").execute(""select 1 + 1"") + ).first()[0] + ) if action == ""this_is_allowed_async"": return True elif action == ""this_is_denied_async"": ``` `pytest -m ds_client` now passes 134 tests. Need to get `pytest -n auto` passing too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1354036967,https://api.github.com/repos/simonw/datasette/issues/1960,1354036967,IC_kwDOBm6k_c5QtPbn,9599,2022-12-16T01:10:12Z,2022-12-16T01:10:12Z,OWNER,"If it does turn out that I can't use `scope=""session""` on this fixture it might not actually be a showstopper: I can take advantage of the fact that `memory_name=""...""` databases stay present in memory for the duration of the process, so I could have it such that each test that uses the `ds_client` fixture DOES construct a fresh `Datasette` instance, but doesn't need to populate the database since they can re-use the in-memory database from the previous object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/issues/1958#issuecomment-1354025319,https://api.github.com/repos/simonw/datasette/issues/1958,1354025319,IC_kwDOBm6k_c5QtMln,11729897,2022-12-16T00:59:12Z,2022-12-16T00:59:12Z,NONE,Awesome. Thank you @simonw.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798, https://github.com/simonw/datasette/issues/1958#issuecomment-1354023960,https://api.github.com/repos/simonw/datasette/issues/1958,1354023960,IC_kwDOBm6k_c5QtMQY,9599,2022-12-16T00:58:14Z,2022-12-16T00:58:19Z,OWNER,"This is in the `0.63.x` branch now, ready to go out in a bug fix release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798, https://github.com/simonw/datasette/issues/1958#issuecomment-1354019543,https://api.github.com/repos/simonw/datasette/issues/1958,1354019543,IC_kwDOBm6k_c5QtLLX,9599,2022-12-16T00:54:11Z,2022-12-16T00:54:11Z,OWNER,"To test the fix, I did a fresh checkout of `simonw/datasette` and edited `Dockerfile` to look like this: ```dockerfile FROM python:3.11.0-slim-bullseye as build RUN apt-get update && \ apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg/info/* RUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \ find /usr/local/lib -name '__pycache__' | xargs rm -r && \ rm -rf /root/.cache/pip EXPOSE 8001 CMD [""datasette""] ``` Then I built it like this: ``` docker build -f Dockerfile -t datasette-pre-click . ``` And ran like this: ``` docker run -p 8001:8001 -v $(pwd):/mnt datasette-pre-click datasette --root -p 8001 -h 0.0.0.0 ``` This exhibited the same problem. Then I pushed a changed to branch and changed the line to: ``` RUN pip install https://github.com/simonw/datasette/archive/refs/heads/click-echo-root.zip && \ ``` Ran this: ``` docker build -f Dockerfile -t datasette-post-click . ``` And this: ``` docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0 ``` It fixed the problem! ``` datasette % docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0 http://0.0.0.0:8001/-/auth-token?token=6542dcf5c8f34f8d13f4af9ce728359c602469efb54029098562bd06c87ad26d INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798, https://github.com/simonw/datasette/issues/1958#issuecomment-1354008688,https://api.github.com/repos/simonw/datasette/issues/1958,1354008688,IC_kwDOBm6k_c5QtIhw,9599,2022-12-16T00:45:40Z,2022-12-16T00:45:40Z,OWNER,The fix may just be to switch to `click.echo()`. https://click.palletsprojects.com/en/8.1.x/api/#click.echo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798, https://github.com/simonw/datasette/issues/1958#issuecomment-1353977605,https://api.github.com/repos/simonw/datasette/issues/1958,1353977605,IC_kwDOBm6k_c5QtA8F,9599,2022-12-16T00:38:23Z,2022-12-16T00:38:23Z,OWNER,"Confirmed, I just got the same result: ``` - % docker run datasetteproject/datasette pip install datasette-upload-csvs ~ % docker commit $(docker ps -lq) datasette-with-plugins sha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f ~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0 INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ^CINFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [1] http://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98 ``` Note how the auth-token URL is only displayed after you hit `Ctrl+C`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798, https://github.com/simonw/datasette/pull/1960#issuecomment-1353812913,https://api.github.com/repos/simonw/datasette/issues/1960,1353812913,IC_kwDOBm6k_c5QsYux,9599,2022-12-15T22:48:54Z,2022-12-15T22:48:54Z,OWNER,"This is all very broken: ``` % pytest -x --pdb ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items tests/test_package.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_cli_serve_get.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_black.py . [ 0%] tests/test_api.py E >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fixturedef = , request = > @pytest.hookimpl(hookwrapper=True) def pytest_fixture_setup( fixturedef: FixtureDef, request: SubRequest ) -> Optional[object]: """"""Adjust the event loop policy when an event loop is produced."""""" if fixturedef.argname == ""event_loop"": outcome = yield > loop = outcome.get_result() /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pytest_asyncio/plugin.py:377: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: in event_loop return asyncio.get_event_loop() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_event_loop(self): """"""Get the event loop for the current context. Returns an instance of EventLoop or raises an exception. """""" if (self._local._loop is None and not self._local._set_called and threading.current_thread() is threading.main_thread()): self.set_event_loop(self.new_event_loop()) if self._local._loop is None: > raise RuntimeError('There is no current event loop in thread %r.' % threading.current_thread().name) E RuntimeError: There is no current event loop in thread 'MainThread'. /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py:656: RuntimeError >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py(656)get_event_loop() -> raise RuntimeError('There is no current event loop in thread %r.' (Pdb) q =================================================================================== warnings summary ==================================================================================== tests/test_cli.py::test_inspect_cli_writes_to_file tests/test_cli.py::test_inspect_cli /Users/simon/Dropbox/Development/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop loop = asyncio.get_event_loop() tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error tests/test_cli_serve_get.py::test_serve_with_get /Users/simon/Dropbox/Development/datasette/datasette/cli.py:596: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(ds.invoke_startup()) tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error tests/test_cli_serve_get.py::test_serve_with_get /Users/simon/Dropbox/Development/datasette/datasette/cli.py:599: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(check_databases(ds)) tests/test_api.py::test_homepage /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: DeprecationWarning: There is no current event loop return asyncio.get_event_loop() -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================================ short test summary info ================================================================================ ERROR tests/test_api.py::test_homepage - RuntimeError: There is no current event loop in thread 'MainThread'. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Quitting debugger !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ======================================================================== 7 passed, 7 warnings, 1 error in 19.15s ======================================================================== (datasette) datasette % ``` The problem looks to be caused by this: https://github.com/simonw/datasette/blob/87737aa1ace82fa7b54c60c41471ec9a661f5299/tests/conftest.py#L28-L30 Which I found necessary in order to have `async def` fixtures that could be shared on the `scope=""session""` basis. Can I work around this, or is `scope=""session""` for async fixtures incompatible with my test suite for some reason? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1353805839,https://api.github.com/repos/simonw/datasette/issues/1960,1353805839,IC_kwDOBm6k_c5QsXAP,9599,2022-12-15T22:38:37Z,2022-12-15T22:38:37Z,OWNER,"I'm going to make `.status_code` work on `TestClient` response too, so I don't have to worry about using both `status` or `status_code` depending on which kind of object I am using.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1353765125,https://api.github.com/repos/simonw/datasette/issues/1960,1353765125,IC_kwDOBm6k_c5QsNEF,9599,2022-12-15T22:00:04Z,2022-12-15T22:00:04Z,OWNER,I'm going to punt on that for the moment and continue to use `app_client` for tests that use that mechanism.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1353763837,https://api.github.com/repos/simonw/datasette/issues/1960,1353763837,IC_kwDOBm6k_c5QsMv9,9599,2022-12-15T21:59:05Z,2022-12-15T21:59:05Z,OWNER,"Here's an annoying error: ``` > response4 = await ds_client.post( ""/-/logout"", csrftoken_from=True, cookies={""ds_actor"": ds_client.actor_cookie({""id"": ""test""})}, ) tests/test_auth.py:88: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , path = '/-/logout' kwargs = {'cookies': {'ds_actor': 'eyJhIjp7ImlkIjoidGVzdCJ9fQ.fuFCTJG5XE-RNnUM7dcnXx9sPvE'}, 'csrftoken_from': True}, client = async def post(self, path, **kwargs): await self.ds.invoke_startup() async with httpx.AsyncClient(app=self.app) as client: > return await client.post(self._fix(path), **kwargs) E TypeError: AsyncClient.post() got an unexpected keyword argument 'csrftoken_from' ``` I need an alternative to the `csrftoken_from` mechanism I built for `TestClient`: https://github.com/simonw/datasette/blob/0b68996cc511b3a801f0cd0157bd66332d75f46f/datasette/utils/testing.py#L77-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/pull/1960#issuecomment-1353749401,https://api.github.com/repos/simonw/datasette/issues/1960,1353749401,IC_kwDOBm6k_c5QsJOZ,9599,2022-12-15T21:47:27Z,2022-12-15T21:47:27Z,OWNER,"I'm using this new mark: ```python @pytest.mark.ds_client ``` Purely so I can run all of the tests that I've refactored using: ``` pytest -m ds_client ``` I'll likely remove this once the test refactoring project is complete.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951, https://github.com/simonw/datasette/issues/1959#issuecomment-1353747370,https://api.github.com/repos/simonw/datasette/issues/1959,1353747370,IC_kwDOBm6k_c5QsIuq,9599,2022-12-15T21:45:14Z,2022-12-15T21:45:14Z,OWNER,I'm going to do this in a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664, https://github.com/simonw/datasette/issues/1959#issuecomment-1353738075,https://api.github.com/repos/simonw/datasette/issues/1959,1353738075,IC_kwDOBm6k_c5QsGdb,9599,2022-12-15T21:35:56Z,2022-12-15T21:35:56Z,OWNER,"I built that `OldResponse` class: ```diff diff --git a/tests/utils.py b/tests/utils.py index 191ead9b..f39ac434 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -30,3 +30,25 @@ def inner_html(soup): def has_load_extension(): conn = sqlite3.connect("":memory:"") return hasattr(conn, ""enable_load_extension"") + + +class OldResponse: + ""Transform an HTTPX response to simulate the older TestClient responses"" + # https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091 + def __init__(self, response): + self.response = response + self._json = None + + @property + def headers(self): + return self.response.headers + + @property + def status(self): + return self.response.status_code + + @property + def json(self): + if self._json is None: + self._json = self.response.json() + return self._json ``` I can use it in tests like this: ```python @pytest.mark.asyncio async def test_homepage(ds_client): response = OldResponse(await ds_client.get(""/.json"")) assert response.status == 200 assert ""application/json; charset=utf-8"" == response.headers[""content-type""] assert response.json.keys() == {""fixtures"": 0}.keys() d = response.json[""fixtures""] assert d[""name""] == ""fixtures"" assert d[""tables_count""] == 24 assert len(d[""tables_and_views_truncated""]) == 5 assert d[""tables_and_views_more""] is True # 4 hidden FTS tables + no_primary_key (hidden in metadata) assert d[""hidden_tables_count""] == 6 # 201 in no_primary_key, plus 6 in other hidden tables: assert d[""hidden_table_rows_sum""] == 207, response.json assert d[""views_count""] == 4 ``` But as I work through the tests I'm finding it's actually not too hard to port them over, so I likely won't use it after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664, https://github.com/simonw/datasette/issues/1959#issuecomment-1353728682,https://api.github.com/repos/simonw/datasette/issues/1959,1353728682,IC_kwDOBm6k_c5QsEKq,9599,2022-12-15T21:28:35Z,2022-12-15T21:28:35Z,OWNER,"Got this error trying to have two tests use the same `ds_client` async fixture when I added `scope=""session""` to that fixture: - https://github.com/tortoise/tortoise-orm/issues/638 Adding this to `conftest.py` (as suggested in that issue thread) seemed to fix it: ```python @pytest.fixture(scope=""session"") def event_loop(): return asyncio.get_event_loop() ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664, https://github.com/simonw/datasette/issues/1619#issuecomment-1353721442,https://api.github.com/repos/simonw/datasette/issues/1619,1353721442,IC_kwDOBm6k_c5QsCZi,2090382,2022-12-15T21:20:53Z,2022-12-15T21:20:53Z,NONE,"i'm also getting bit by this. I'm trying to set up an nginx reverse proxy in front of multiple datasette backends. When I run it locally or behind the proxy, I see the `base_url` value added a second time to the path for various action links on table pages (view as JSON, sort by column, etc).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414, https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091,https://api.github.com/repos/simonw/datasette/issues/1959,1353721091,IC_kwDOBm6k_c5QsCUD,9599,2022-12-15T21:20:32Z,2022-12-15T21:20:32Z,OWNER,Rather than tediously rewriting every single test to the new shape I'm going to try a wrapper for that HTTPX response that transforms it into an imitation of the one returned by the existing `TestClient` class.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664, https://github.com/simonw/datasette/issues/1959#issuecomment-1353720559,https://api.github.com/repos/simonw/datasette/issues/1959,1353720559,IC_kwDOBm6k_c5QsCLv,9599,2022-12-15T21:19:56Z,2022-12-15T21:19:56Z,OWNER,"Here's a port of the first `def ...(app_client)` test. Note that the TestClient object works slightly differently from the HTTPX response returned by `await datasette.client.get(...)`: ```diff diff --git a/datasette/app.py b/datasette/app.py index f3cb8876..b770b469 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -281,7 +281,7 @@ class Datasette: raise self.crossdb = crossdb self.nolock = nolock - if memory or crossdb or not self.files: + if memory or crossdb or (not self.files and memory is not False): self.add_database( Database(self, is_mutable=False, is_memory=True), name=""_memory"" ) diff --git a/pytest.ini b/pytest.ini index 559e518c..0bcb0d1e 100644 --- a/pytest.ini +++ b/pytest.ini @@ -8,4 +8,5 @@ filterwarnings= ignore:.*current_task.*:PendingDeprecationWarning markers = serial: tests to avoid using with pytest-xdist + ds_client: tests using the ds_client fixture asyncio_mode = strict diff --git a/tests/conftest.py b/tests/conftest.py index cd735e12..648423ba 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -2,6 +2,7 @@ import httpx import os import pathlib import pytest +import pytest_asyncio import re import subprocess import tempfile @@ -23,6 +24,22 @@ UNDOCUMENTED_PERMISSIONS = { } +@pytest_asyncio.fixture +async def ds_client(): + from datasette.app import Datasette + from .fixtures import METADATA, PLUGINS_DIR + ds = Datasette(memory=False, metadata=METADATA, plugins_dir=PLUGINS_DIR) + from .fixtures import TABLES, TABLE_PARAMETERIZED_SQL + db = ds.add_memory_database(""fixtures"") + def prepare(conn): + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) + await db.execute_write_fn(prepare) + return ds.client + + def pytest_report_header(config): return ""SQLite: {}"".format( sqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchone()[0] diff --git a/tests/test_api.py b/tests/test_api.py index 5f2a6ea6..ddf4219c 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -23,12 +23,15 @@ import sys import urllib -def test_homepage(app_client): - response = app_client.get(""/.json"") - assert response.status == 200 +@pytest.mark.ds_client +@pytest.mark.asyncio +async def test_homepage(ds_client): + response = await ds_client.get(""/.json"") + assert response.status_code == 200 assert ""application/json; charset=utf-8"" == response.headers[""content-type""] - assert response.json.keys() == {""fixtures"": 0}.keys() - d = response.json[""fixtures""] + data = response.json() + assert data.keys() == {""fixtures"": 0}.keys() + d = data[""fixtures""] assert d[""name""] == ""fixtures"" assert d[""tables_count""] == 24 assert len(d[""tables_and_views_truncated""]) == 5 @@ -36,7 +39,7 @@ def test_homepage(app_client): # 4 hidden FTS tables + no_primary_key (hidden in metadata) assert d[""hidden_tables_count""] == 6 # 201 in no_primary_key, plus 6 in other hidden tables: - assert d[""hidden_table_rows_sum""] == 207, response.json + assert d[""hidden_table_rows_sum""] == 207, data assert d[""views_count""] == 4 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664, https://github.com/simonw/datasette/issues/1959#issuecomment-1353707828,https://api.github.com/repos/simonw/datasette/issues/1959,1353707828,IC_kwDOBm6k_c5Qr_E0,9599,2022-12-15T21:06:29Z,2022-12-15T21:06:29Z,OWNER,"Previous, abandoned attempt at this work (for #1843): ```diff diff --git a/datasette/app.py b/datasette/app.py index 7e682498..cf35c3a2 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -228,7 +228,7 @@ class Datasette: template_dir=None, plugins_dir=None, static_mounts=None, - memory=False, + memory=None, settings=None, secret=None, version_note=None, @@ -238,6 +238,7 @@ class Datasette: nolock=False, ): self._startup_invoked = False + self._extra_on_startup = [] assert config_dir is None or isinstance( config_dir, Path ), ""config_dir= should be a pathlib.Path"" @@ -278,7 +279,7 @@ class Datasette: raise self.crossdb = crossdb self.nolock = nolock - if memory or crossdb or not self.files: + if memory or crossdb or (not self.files and memory is not False): self.add_database( Database(self, is_mutable=False, is_memory=True), name=""_memory"" ) @@ -391,6 +392,9 @@ class Datasette: self._root_token = secrets.token_hex(32) self.client = DatasetteClient(self) + def _add_on_startup(self, fn): + self._extra_on_startup.append(fn) + async def refresh_schemas(self): if self._refresh_schemas_lock.locked(): return @@ -431,6 +435,8 @@ class Datasette: # This must be called for Datasette to be in a usable state if self._startup_invoked: return + for fn in self._extra_on_startup: + await fn() # Register permissions, but watch out for duplicate name/abbr names = {} abbrs = {} @@ -1431,9 +1437,9 @@ class Datasette: ) if self.setting(""trace_debug""): asgi = AsgiTracer(asgi) - asgi = AsgiRunOnFirstRequest(asgi, on_startup=[setup_db, self.invoke_startup]) for wrapper in pm.hook.asgi_wrapper(datasette=self): asgi = wrapper(asgi) + asgi = AsgiRunOnFirstRequest(asgi, on_startup=[setup_db, self.invoke_startup]) return asgi diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py index 56690251..986755cb 100644 --- a/datasette/utils/asgi.py +++ b/datasette/utils/asgi.py @@ -423,9 +423,9 @@ class AsgiFileDownload: class AsgiRunOnFirstRequest: - def __init__(self, asgi, on_startup): + def __init__(self, app, on_startup): assert isinstance(on_startup, list) - self.asgi = asgi + self.app = app self.on_startup = on_startup self._started = False @@ -434,4 +434,4 @@ class AsgiRunOnFirstRequest: self._started = True for hook in self.on_startup: await hook() - return await self.asgi(scope, receive, send) + return await self.app(scope, receive, send) diff --git a/tests/conftest.py b/tests/conftest.py index cd735e12..d1301943 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -23,6 +23,15 @@ UNDOCUMENTED_PERMISSIONS = { } +# @pytest.fixture(autouse=True) +# def log_name_of_test_before_test(request): +# # To help identify tests that are hanging +# name = str(request.node) +# with open(""/tmp/test.log"", ""a"") as f: +# f.write(name + ""\n"") +# yield + + def pytest_report_header(config): return ""SQLite: {}"".format( sqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchone()[0] diff --git a/tests/fixtures.py b/tests/fixtures.py index a6700239..18d3f1b7 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -101,6 +101,19 @@ EXPECTED_PLUGINS = [ ] +def _populate_connection(conn): + # Drop any tables and views that exist + to_drop = conn.execute( + ""SELECT name, type FROM sqlite_master where type in ('table', 'view')"" + ).fetchall() + for name, type in to_drop: + conn.execute(f""DROP {type} IF EXISTS [{name}]"") + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) + + @contextlib.contextmanager def make_app_client( sql_time_limit_ms=None, @@ -117,45 +130,22 @@ def make_app_client( metadata=None, crossdb=False, ): - with tempfile.TemporaryDirectory() as tmpdir: - filepath = os.path.join(tmpdir, filename) - if is_immutable: - files = [] - immutables = [filepath] - else: - files = [filepath] - immutables = [] - conn = sqlite3.connect(filepath) - conn.executescript(TABLES) - for sql, params in TABLE_PARAMETERIZED_SQL: - with conn: - conn.execute(sql, params) - # Close the connection to avoid ""too many open files"" errors - conn.close() - if extra_databases is not None: - for extra_filename, extra_sql in extra_databases.items(): - extra_filepath = os.path.join(tmpdir, extra_filename) - c2 = sqlite3.connect(extra_filepath) - c2.executescript(extra_sql) - c2.close() - # Insert at start to help test /-/databases ordering: - files.insert(0, extra_filepath) - os.chdir(os.path.dirname(filepath)) - settings = settings or {} - for key, value in { - ""default_page_size"": 50, - ""max_returned_rows"": max_returned_rows or 100, - ""sql_time_limit_ms"": sql_time_limit_ms or 200, - # Default is 3 but this results in ""too many open files"" - # errors when running the full test suite: - ""num_sql_threads"": 1, - }.items(): - if key not in settings: - settings[key] = value + settings = settings or {} + for key, value in { + ""default_page_size"": 50, + ""max_returned_rows"": max_returned_rows or 100, + ""sql_time_limit_ms"": sql_time_limit_ms or 200, + # Default is 3 but this results in ""too many open files"" + # errors when running the full test suite: + ""num_sql_threads"": 1, + }.items(): + if key not in settings: + settings[key] = value + # We can use an in-memory database, but only if we're not doing anything + # with is_immutable or extra_databases and filename is the default + if not is_immutable and not extra_databases and filename == ""fixtures.db"": ds = Datasette( - files, - immutables=immutables, - memory=memory, + memory=memory or False, cors=cors, metadata=metadata or METADATA, plugins_dir=PLUGINS_DIR, @@ -165,12 +155,57 @@ def make_app_client( template_dir=template_dir, crossdb=crossdb, ) + db = ds.add_memory_database(""fixtures"") + + async def populate_fixtures(): + print(""Here we go... populating fixtures"") + await db.execute_write_fn(_populate_connection) + + ds._add_on_startup(populate_fixtures) yield TestClient(ds) - # Close as many database connections as possible - # to try and avoid too many open files error - for db in ds.databases.values(): - if not db.is_memory: - db.close() + else: + with tempfile.TemporaryDirectory() as tmpdir: + filepath = os.path.join(tmpdir, filename) + if is_immutable: + files = [] + immutables = [filepath] + else: + files = [filepath] + immutables = [] + + conn = sqlite3.connect(filepath) + _populate_connection(conn) + # Close the connection to reduce ""too many open files"" errors + conn.close() + + if extra_databases is not None: + for extra_filename, extra_sql in extra_databases.items(): + extra_filepath = os.path.join(tmpdir, extra_filename) + c2 = sqlite3.connect(extra_filepath) + c2.executescript(extra_sql) + c2.close() + # Insert at start to help test /-/databases ordering: + files.insert(0, extra_filepath) + os.chdir(os.path.dirname(filepath)) + ds = Datasette( + files, + immutables=immutables, + memory=memory, + cors=cors, + metadata=metadata or METADATA, + plugins_dir=PLUGINS_DIR, + settings=settings, + inspect_data=inspect_data, + static_mounts=static_mounts, + template_dir=template_dir, + crossdb=crossdb, + ) + yield TestClient(ds) + # Close as many database connections as possible + # to try and avoid too many open files error + for db in ds.databases.values(): + if not db.is_memory: + db.close() @pytest.fixture(scope=""session"") diff --git a/tests/test_cli.py b/tests/test_cli.py index d3e015fa..d9e4e457 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,5 +1,6 @@ from .fixtures import ( app_client, + app_client_with_cors, make_app_client, TestClient as _TestClient, EXPECTED_PLUGINS, @@ -38,7 +39,7 @@ def test_inspect_cli(app_client): assert expected_count == database[""tables""][table_name][""count""] -def test_inspect_cli_writes_to_file(app_client): +def test_inspect_cli_writes_to_file(app_client_with_cors): runner = CliRunner() result = runner.invoke( cli, [""inspect"", ""fixtures.db"", ""--inspect-file"", ""foo.json""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664, https://github.com/simonw/datasette/issues/1959#issuecomment-1353705072,https://api.github.com/repos/simonw/datasette/issues/1959,1353705072,IC_kwDOBm6k_c5Qr-Zw,9599,2022-12-15T21:04:07Z,2022-12-15T21:04:07Z,OWNER,I'm going to start by getting every test that uses the raw `(app_client)` fixture and nothing else (194 at the moment) to switch to `async def` using a shared Datasette instance and `datasette.client.get()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664, https://github.com/simonw/datasette/issues/1955#issuecomment-1353701674,https://api.github.com/repos/simonw/datasette/issues/1955,1353701674,IC_kwDOBm6k_c5Qr9kq,9599,2022-12-15T21:00:51Z,2022-12-15T21:00:51Z,OWNER,"OK, I've broken the test suite here. I'm going to revert these two commits: - https://github.com/simonw/datasette/commit/dc18f62089e5672d03176f217d7840cdafa5c447 - https://github.com/simonw/datasette/commit/51ee8caa4a697fa3f4120e93b1c205b714a6cdc7 Then I'll do a bunch of work making the test suite more robust before I try this again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353694582,https://api.github.com/repos/simonw/datasette/issues/1955,1353694582,IC_kwDOBm6k_c5Qr712,9599,2022-12-15T20:52:46Z,2022-12-15T20:52:46Z,OWNER,"Just noticed this: https://github.com/simonw/datasette/actions/runs/3706504228/jobs/6281796135 This suggests that the regular tests passed in CI fine, but the non-serial ones failed. I'm going to try running everything using `pytest -n auto` without splitting serial and non-serial tests. Maybe the serial thing isn't needed any more?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1843#issuecomment-1353690591,https://api.github.com/repos/simonw/datasette/issues/1843,1353690591,IC_kwDOBm6k_c5Qr63f,9599,2022-12-15T20:49:05Z,2022-12-15T20:49:05Z,OWNER,"I have a nasty feeling the cleaner solution for this would involve porting my entire test suite from `def test_blah(app_client)` sync functions (which work due to a `@async_to_sync` call in `TestClient`) to `async def test_blah(async_fixture):` functions instead. I've been using that latter pattern for new tests (and plugin tests) for quite a while now, but I never took on the job of refactoring all of the old ones. A search for `(app_client):` across the whole project currently returns 194 results which might be a reasonable target to try switching to the new pattern as a starting point. No idea if it will have much impact on the ""Too many open files"" errors though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1955#issuecomment-1353683238,https://api.github.com/repos/simonw/datasette/issues/1955,1353683238,IC_kwDOBm6k_c5Qr5Em,9599,2022-12-15T20:42:18Z,2022-12-15T20:42:18Z,OWNER,"Possibly related issue: - https://github.com/pytest-dev/pytest-xdist/issues/60","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353680261,https://api.github.com/repos/simonw/datasette/issues/1955,1353680261,IC_kwDOBm6k_c5Qr4WF,9599,2022-12-15T20:39:19Z,2022-12-15T20:39:19Z,OWNER,"When I hit `Ctr+C` here's the traceback I get: ``` ^C^CException ignored in: Traceback (most recent call last): File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py"", line 1530, in _shutdown !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py:324: KeyboardInterrupt (to show a full traceback on KeyboardInterrupt use --full-trace) Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/pytest"", line 8, in atexit_call() File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/concurrent/futures/thread.py"", line 31, in _python_exit sys.exit(console_main()) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py"", line 187, in console_main t.join() File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py"", line 1089, in join self._wait_for_tstate_lock() File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py"", line 1109, in _wait_for_tstate_lock if lock.acquire(block, timeout): KeyboardInterrupt: code = main() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py"", line 164, in main ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main( File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py"", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py"", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py"", line 60, in _multicall return outcome.get_result() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_result.py"", line 60, in get_result raise ex[1].with_traceback(ex[2]) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py"", line 39, in _multicall res = hook_impl.function(*args) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/main.py"", line 315, in pytest_cmdline_main return wrap_session(config, _main) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/main.py"", line 303, in wrap_session config.hook.pytest_sessionfinish( File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py"", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py"", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py"", line 55, in _multicall gen.send(outcome) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/terminal.py"", line 798, in pytest_sessionfinish outcome.get_result() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_result.py"", line 60, in get_result raise ex[1].with_traceback(ex[2]) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py"", line 39, in _multicall res = hook_impl.function(*args) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/xdist/dsession.py"", line 88, in pytest_sessionfinish nm.teardown_nodes() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/xdist/workermanage.py"", line 79, in teardown_nodes self.group.terminate(self.EXIT_TIMEOUT) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/multi.py"", line 215, in terminate safe_terminate( File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/multi.py"", line 311, in safe_terminate reply.get() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/gateway_base.py"", line 206, in get self.waitfinish(timeout) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/gateway_base.py"", line 213, in waitfinish if not self._result_ready.wait(timeout): File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py"", line 600, in wait signaled = self._cond.wait(timeout) File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py"", line 320, in wait waiter.acquire() KeyboardInterrupt ``` It looks to me like this relates to `pytest-xdist` istelf - it's waiting on some locks but `site-packages/xdist/workermanage.py` shows up in that track.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1843#issuecomment-1353522652,https://api.github.com/repos/simonw/datasette/issues/1843,1353522652,IC_kwDOBm6k_c5QrR3c,9599,2022-12-15T18:21:27Z,2022-12-15T18:21:27Z,OWNER,"I'll still use on-disk test databases for `is_immutable=True`, but not for the majority of tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1353522211,https://api.github.com/repos/simonw/datasette/issues/1843,1353522211,IC_kwDOBm6k_c5QrRwj,9599,2022-12-15T18:21:02Z,2022-12-15T18:21:02Z,OWNER,"When I initially built this test suite Datasette didn't have the `memory_name=` mechanism for creating persistent in-memory databases. I'm going to see if I can switch to that for the majority of Datasette's tests. Might find that doing so both fixes this ""too many open files"" issue AND gives me a significant speed improvement to the test site too! Hopefully I can do most of the work on that in this big ugly function: https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/fixtures.py#L104-L173","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1843#issuecomment-1353520615,https://api.github.com/repos/simonw/datasette/issues/1843,1353520615,IC_kwDOBm6k_c5QrRXn,9599,2022-12-15T18:19:25Z,2022-12-15T18:19:25Z,OWNER,"I've been seeing this error again: ``` ERROR tests/test_api_write.py::test_create_table[input16-400-expected_response16] - OSError: [Errno 24] Too ... ERROR tests/test_api_write.py::test_create_table[input17-400-expected_response17] - OSError: [Errno 24] Too ... ERROR tests/test_api_write.py::test_create_table[input18-400-expected_response18] - OSError: [Errno 24] Too ... ``` It doesn't happen in CI, and it turns out that's because CI runs `pytest -n auto` which splits the tests across multiple parallel processes. I've been working around the error on my laptop using `pytest -n auto` there too, but I'd rather not have to do that. This is also getting in my way when I try to debug other issues, like this one: - #1955","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705, https://github.com/simonw/datasette/issues/1955#issuecomment-1353516572,https://api.github.com/repos/simonw/datasette/issues/1955,1353516572,IC_kwDOBm6k_c5QrQYc,9599,2022-12-15T18:15:28Z,2022-12-15T18:15:28Z,OWNER,"I added `return` to the first line of that test to disable it, then ran again - and now it's hanging at about the same progress point through the tests but in a different test: ![Image](https://user-images.githubusercontent.com/9599/207936587-30ebf780-c0da-4e62-b20b-e274e0adaa19.png) So this time it was hanging at `test_urlsafe_components()`. So it's clearly not the individual tests themselves that are the problem - something about running the entire test suite in one go is incompatible with this change for some reason.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353512099,https://api.github.com/repos/simonw/datasette/issues/1955,1353512099,IC_kwDOBm6k_c5QrPSj,9599,2022-12-15T18:11:27Z,2022-12-15T18:11:27Z,OWNER,"This is surprising! ![Image](https://user-images.githubusercontent.com/9599/207935885-e1f51983-0621-4490-86a6-fafd4c876f41.png) The logs suggest that the test suite hung running this test here: https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/test_utils.py#L55-L58 I find that very hard to believe.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353509776,https://api.github.com/repos/simonw/datasette/issues/1955,1353509776,IC_kwDOBm6k_c5QrOuQ,9599,2022-12-15T18:09:26Z,2022-12-15T18:09:26Z,OWNER,"I added this to `conftest.py`: ```python @pytest.fixture(autouse=True) def log_name_of_test_before_test(request): # To help identify tests that are hanging name = str(request.node) with open(""/tmp/test.log"", ""a"") as f: f.write(name + ""\n"") yield ``` This logs out the name of each test to `/tmp/test.log` before running the test - so I can wait until it hangs and see which test it was that caused that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353473571,https://api.github.com/repos/simonw/datasette/issues/1955,1353473571,IC_kwDOBm6k_c5QrF4j,9599,2022-12-15T17:43:28Z,2022-12-15T17:43:48Z,OWNER,"Running: pytest -n auto -x -v On may laptop to see if I can replicate.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353473086,https://api.github.com/repos/simonw/datasette/issues/1955,1353473086,IC_kwDOBm6k_c5QrFw-,9599,2022-12-15T17:43:08Z,2022-12-15T17:43:08Z,OWNER,It looks like that fix _almost_ works... except it seems to push the tests into an infinite loop or similar? They're not finishing their runs from what I can see.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353448095,https://api.github.com/repos/simonw/datasette/issues/1955,1353448095,IC_kwDOBm6k_c5Qq_qf,9599,2022-12-15T17:25:05Z,2022-12-15T17:25:05Z,OWNER,"So actually that `setup_db()` function I wrote back in 2019 has not been executing for most of Datasette's tests. Which seems bad. I'm inclined to ditch `AsgiLifespan` entirely in favour of the mechanism I described above, where `invoke_startup()` is called for every request on the first request processed by the server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353443718,https://api.github.com/repos/simonw/datasette/issues/1955,1353443718,IC_kwDOBm6k_c5Qq-mG,9599,2022-12-15T17:23:12Z,2022-12-15T17:23:55Z,OWNER,"That may not be the best fix here. It turns out this pattern: ```python async def get(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.get(self._fix(path), **kwargs) ``` Doesn't trigger that `AsgiLifespan` class. I wrote about that previously in this TIL: https://til.simonwillison.net/asgi/lifespan-test-httpx","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353423584,https://api.github.com/repos/simonw/datasette/issues/1955,1353423584,IC_kwDOBm6k_c5Qq5rg,9599,2022-12-15T17:13:18Z,2022-12-15T17:22:59Z,OWNER,"Wow, just spotted this in the code - it turns out I solved this problem a different (and better) way long before i introduced `invoke_startup()`! https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1416-L1440","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1352674924,https://api.github.com/repos/simonw/datasette/issues/1955,1352674924,IC_kwDOBm6k_c5QoC5s,9599,2022-12-15T07:46:36Z,2022-12-15T07:46:36Z,OWNER,"It's possible the fix for this might be for the first incoming HTTP request to trigger `invoke_startup()` if it hasn't been called yet - similar to the hack I put in place for `datasette.client.get()` in tests: https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1728-L1731 This would be a much more elegant fix, I could remove those multiple `invoke_startup()` calls entirely - and remove this tip from the documentation too: https://docs.datasette.io/en/0.63.2/testing_plugins.html#setting-up-a-datasette-test-instance","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1958#issuecomment-1352644281,https://api.github.com/repos/simonw/datasette/issues/1958,1352644281,IC_kwDOBm6k_c5Qn7a5,9599,2022-12-15T07:08:14Z,2022-12-15T07:08:14Z,OWNER,Thanks for the details write-up! This looks like a bug in Datasette itself when run with Docker. Moving this issue there.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798, https://github.com/simonw/datasette/issues/1955#issuecomment-1352643333,https://api.github.com/repos/simonw/datasette/issues/1955,1352643333,IC_kwDOBm6k_c5Qn7MF,9599,2022-12-15T07:07:29Z,2022-12-15T07:07:29Z,OWNER,"Datasette 0.63 is the release that broke this, thanks to this issue: - https://github.com/simonw/datasette/issues/1809","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1352643049,https://api.github.com/repos/simonw/datasette/issues/1955,1352643049,IC_kwDOBm6k_c5Qn7Hp,9599,2022-12-15T07:07:10Z,2022-12-15T07:07:10Z,OWNER,"This is definitely a regression: Datasette is meant to work in those environments, and I didn't think to test them when I added the `invoke_startup()` hook. Coincidentally I actually built a plugin for running Datasette with Gunicorn just a couple of months ago: https://datasette.io/plugins/datasette-gunicorn And I just tested and it has the same bug you describe here! Filed: - https://github.com/simonw/datasette-gunicorn/issues/5 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1953#issuecomment-1352459146,https://api.github.com/repos/simonw/datasette/issues/1953,1352459146,IC_kwDOBm6k_c5QnOOK,9599,2022-12-15T02:02:15Z,2022-12-15T02:02:15Z,OWNER,"``` The third Datasette 1.0 alpha release adds upsert support to the JSON API, plus the ability to specify finely grained permissions when creating an API token. - New `/db/table/-/upsert` API, [documented here](https://docs.datasette.io/en/latest/json_api.html#tableupsertview). upsert is an update-or-replace: existing rows will have specified keys updated, but if no row matches the incoming primary key a brand new row will be inserted instead. ([#1878](https://github.com/simonw/datasette/issues/1878)) - New [register_permissions(datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-register-permissions) plugin hook. Plugins can now register named permissions, which will then be listed in various interfaces that show available permissions. ([#1940](https://github.com/simonw/datasette/issues/1940)) - The `/db/-/create` API for [creating a table](https://docs.datasette.io/en/latest/json_api.html#tablecreateview) now accepts `""ignore"": true` and `""replace"": true` options when called with the `""rows""` property that creates a new table based on an example set of rows. This means the API can be called multiple times with different rows, setting rules for what should happen if a primary key collides with an existing row. ([#1927](https://github.com/simonw/datasette/issues/1927)) - Arbitrary permissions can now be configured at the instance, database and resource (table, SQL view or canned query) level in Datasette's [Metadata](https://docs.datasette.io/en/latest/metadata.html#metadata) JSON and YAML files. The new `""permissions""` key can be used to specify which actors should have which permissions. See [Other permissions in metadata](https://docs.datasette.io/en/latest/authentication.html#authentication-permissions-other) for details. ([#1636](https://github.com/simonw/datasette/issues/1636)) - The `/-/create-token` page can now be used to create API tokens which are restricted to just a subset of actions, including against specific databases or resources. See [API Tokens](https://docs.datasette.io/en/latest/authentication.html#createtokenview) for details. ([#1947](https://github.com/simonw/datasette/issues/1947)) - Likewise, the `datasette create-token` CLI command can now create tokens with [a subset of permissions](https://docs.datasette.io/en/latest/authentication.html#authentication-cli-create-token-restrict). ([#1855](https://github.com/simonw/datasette/issues/1855)) - New datasette.create_token() API method ` for programmatically creating signed API tokens. ([#1951](https://github.com/simonw/datasette/issues/1951)) - `/db/-/create` API now requires actor to have `insert-row` permission in order to use the `""row""` or `""rows""` properties. ([#1937](https://github.com/simonw/datasette/issues/1937)) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495821607, https://github.com/simonw/datasette/issues/1949#issuecomment-1352411327,https://api.github.com/repos/simonw/datasette/issues/1949,1352411327,IC_kwDOBm6k_c5QnCi_,9599,2022-12-15T00:46:27Z,2022-12-15T00:46:27Z,OWNER,"I got this far: ```diff diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index 8b7e83e3..31d41e00 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -54,7 +54,17 @@ def handle_exception(datasette, request, exception): headers = {} if datasette.cors: add_cors_headers(headers) - if request.path.split(""?"")[0].endswith("".json""): + # Return JSON error under certain conditions + should_return_json = ( + # URL ends in .json + request.path.split(""?"")[0].endswith("".json"") + or + # Hints from incoming request headers + request.headers.get(""content-type"") == ""application/json"" + or ""application/json"" in request.headers.get(""accept"", """") + ) + breakpoint() + if should_return_json: return Response.json(info, status=status, headers=headers) else: template = datasette.jinja_env.select_template(templates) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index f27d143f..982543a6 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1140,6 +1140,38 @@ async def test_create_table_permissions( assert data[""errors""] == expected_errors +@pytest.mark.asyncio +@pytest.mark.parametrize( + ""headers,expect_json"", + ( + ({}, False), + ({""Accept"": ""text/html""}, True), + ({""Accept"": ""application/json""}, True), + ({""Content-Type"": ""application/json""}, True), + ({""Accept"": ""application/json, text/plain, */*""}, True), + ({""Content-Type"": ""application/json""}, True), + ({""accept"": ""application/json, text/plain, */*""}, True), + ({""content-type"": ""application/json""}, True), + ), +) +async def test_permission_errors_html_and_json(ds_write, headers, expect_json): + request_headers = {""Authorization"": ""Bearer bad_token""} + request_headers.update(headers) + response = await ds_write.client.post( + ""/data/-/create"", + json={}, + headers=request_headers, + ) + assert response.status_code == 403 + if expect_json: + data = response.json() + assert data[""ok""] is False + assert data[""errors""] == [""Permission denied""] + else: + assert response.headers[""Content-Type""] == ""text/html; charset=utf-8"" + assert ""Permission denied"" in response.text + + @pytest.mark.asyncio @pytest.mark.parametrize( ""input,expected_rows_after"", ``` Then decided I would punt this until the next milestone.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493471221, https://github.com/simonw/datasette/issues/1953#issuecomment-1352410078,https://api.github.com/repos/simonw/datasette/issues/1953,1352410078,IC_kwDOBm6k_c5QnCPe,9599,2022-12-15T00:44:56Z,2022-12-15T00:44:56Z,OWNER,"Highlights: - `/db/table/-/upsert` - ignore and replace for `/db/-/create` - `register_permissions()` plugin hook - `datasette create-token` can create restricted tokens - `/-/create-token` can too - `datasette --get --token` option - `datasette.create_token()` API method Plus some smaller things.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495821607, https://github.com/simonw/datasette/issues/1949#issuecomment-1352378370,https://api.github.com/repos/simonw/datasette/issues/1949,1352378370,IC_kwDOBm6k_c5Qm6gC,9599,2022-12-15T00:02:08Z,2022-12-15T00:04:54Z,OWNER,"I fixed this issue to help research this further: - https://github.com/simonw/datasette-ripgrep/issues/26 Now this search works: I wish I had this feature! - https://github.com/simonw/datasette-ripgrep/issues/24 Looks like I have both `_error()` and `_errors()` functions in there! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493471221, https://github.com/simonw/datasette/issues/1957#issuecomment-1352371019,https://api.github.com/repos/simonw/datasette/issues/1957,1352371019,IC_kwDOBm6k_c5Qm4tL,9599,2022-12-14T23:50:50Z,2022-12-14T23:50:50Z,OWNER,"One option: if any rows were truncated, show a button you can click to run the query again with truncation disabled - maybe with `?_truncate=off` in the URL or similar. But... still want to truncate if the user runs a query that would return multiple MBs of HTML (assuming the other Datasette query limits don't stop that from happening).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497577017, https://github.com/simonw/datasette/issues/1295#issuecomment-1352357322,https://api.github.com/repos/simonw/datasette/issues/1295,1352357322,IC_kwDOBm6k_c5Qm1XK,9599,2022-12-14T23:28:49Z,2022-12-14T23:28:49Z,OWNER,"Related: - #1875","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",855296937, https://github.com/simonw/datasette/issues/1949#issuecomment-1352356356,https://api.github.com/repos/simonw/datasette/issues/1949,1352356356,IC_kwDOBm6k_c5Qm1IE,9599,2022-12-14T23:27:25Z,2022-12-14T23:28:16Z,OWNER,"Also weird: errors returned by that mechanism look like this: ```json { ""ok"": false, ""errors"": [""list of error messages""] } ``` While errors returned by the rest of Datasette look like this: https://latest.datasette.io/fixtures/no_table.json ```json { ""ok"": false, ""error"": ""Table not found: no_table"", ""status"": 404, ""title"": null } ``` Related: - #1875","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493471221, https://github.com/simonw/datasette/issues/1949#issuecomment-1352354927,https://api.github.com/repos/simonw/datasette/issues/1949,1352354927,IC_kwDOBm6k_c5Qm0xv,9599,2022-12-14T23:25:06Z,2022-12-14T23:25:14Z,OWNER,"Looks like the code I've written for permission checking on `TableCreateView` and friends doesn't use the regular `raise Forbidden` or `raise DatasetteError` mechanisms - it does its own thing here: https://github.com/simonw/datasette/blob/9ad76d279e2c3874ca5070626a25458ce129f126/datasette/views/database.py#L580-L584 Which uses this: https://github.com/simonw/datasette/blob/9ad76d279e2c3874ca5070626a25458ce129f126/datasette/views/base.py#L547-L548 Having two different patterns to return errors is bad, I should fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493471221, https://github.com/simonw/datasette/issues/1949#issuecomment-1352340518,https://api.github.com/repos/simonw/datasette/issues/1949,1352340518,IC_kwDOBm6k_c5QmxQm,9599,2022-12-14T23:07:01Z,2022-12-14T23:07:01Z,OWNER,"Easiest fix would be to look for `accept: application/json` and/or `content-type: application/json` headers. Not bullet-proof, so people might occasionally make JSON requests and get back an HTML error - but the documentation can tell people that they need to send those headers if they want to reliably get back JSON error messages. I'm happy with this as a solution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493471221, https://github.com/simonw/datasette/issues/1949#issuecomment-1352338620,https://api.github.com/repos/simonw/datasette/issues/1949,1352338620,IC_kwDOBm6k_c5Qmwy8,9599,2022-12-14T23:05:17Z,2022-12-14T23:05:17Z,OWNER,"Sniffing for a `{` is a little bit tricky though, as the post body is lazily loaded on request here: https://github.com/simonw/datasette/blob/9ad76d279e2c3874ca5070626a25458ce129f126/datasette/utils/asgi.py#L127-L135","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493471221, https://github.com/simonw/datasette/issues/1949#issuecomment-1352335503,https://api.github.com/repos/simonw/datasette/issues/1949,1352335503,IC_kwDOBm6k_c5QmwCP,9599,2022-12-14T23:03:28Z,2022-12-14T23:03:28Z,OWNER,"This raises a more complicated issue At some point I'm likely to want to add an HTML interface for creating tables and inserting and updating rows. The obvious URLs for that are the same as for the JSON API: `/db/table/-/insert` and suchlike. Those endpoints are currently POST only - and can return JSON all the time. If they start accepting form POSTs too they'll need to be able to accept form-encoded data and return HTML instead. That's OK - they can detect incoming JSON thanks to the `content-type` header an the fact that the request body starts with `{` - but the `should_return_json` fix described above could intefere with how errors are returned if I'm not careful. I think it can still work though: I'll only set `should_return_json = True` if the endpoint gets a POST with a body starting `{`, or a content-type JSON header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493471221, https://github.com/simonw/datasette/issues/1949#issuecomment-1352331314,https://api.github.com/repos/simonw/datasette/issues/1949,1352331314,IC_kwDOBm6k_c5QmvAy,9599,2022-12-14T22:59:36Z,2022-12-14T22:59:36Z,OWNER,I'm going to prototype that up to see what it looks like.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493471221, https://github.com/simonw/datasette/issues/1949#issuecomment-1352330825,https://api.github.com/repos/simonw/datasette/issues/1949,1352330825,IC_kwDOBm6k_c5Qmu5J,9599,2022-12-14T22:58:51Z,2022-12-14T22:59:27Z,OWNER,"I need a way for those JSON endpoints to communicate back to the `handle_exception` handler that they are returning JSON, so it knows to behave differently. Since it gets the `request` object, one way could be to have view code set `request.should_return_json = True` so that the handler knows to do something different. It's a bit of a cludge though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493471221, https://github.com/simonw/datasette/issues/1949#issuecomment-1352329027,https://api.github.com/repos/simonw/datasette/issues/1949,1352329027,IC_kwDOBm6k_c5QmudD,9599,2022-12-14T22:56:24Z,2022-12-14T22:57:19Z,OWNER,"Most `.json` errors DO return as JSON, thanks to this: https://github.com/simonw/datasette/blob/c094dde3ff2bae030f261e6440d4fb082eb860a9/datasette/handle_exception.py#L19-L24 https://github.com/simonw/datasette/blob/c094dde3ff2bae030f261e6440d4fb082eb860a9/datasette/handle_exception.py#L57-L58 But that code triggers when the URL ends with `.json` - and none of the JSON write API endpoints (things like `/db/-/create` and `/db/table/-/insert`) follow that convention.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493471221, https://github.com/simonw/datasette/issues/1956#issuecomment-1352075845,https://api.github.com/repos/simonw/datasette/issues/1956,1352075845,IC_kwDOBm6k_c5QlwpF,9599,2022-12-14T19:57:17Z,2022-12-14T19:58:22Z,OWNER,"I'm going to test this using calls to `ds.permission_allowed()` with an actor with `_r` block. I can add extra tests to https://github.com/simonw/datasette/blob/1a3dcf494376e32f7cff110c86a88e5b0a3f3924/tests/test_permissions.py#L605-L616","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497288666, https://github.com/simonw/datasette/issues/1956#issuecomment-1352070655,https://api.github.com/repos/simonw/datasette/issues/1956,1352070655,IC_kwDOBm6k_c5QlvX_,9599,2022-12-14T19:54:36Z,2022-12-14T19:54:36Z,OWNER,Also this code should work with non-abbreviations too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497288666, https://github.com/simonw/datasette/issues/1958#issuecomment-1352644276,https://api.github.com/repos/simonw/datasette/issues/1958,1352644276,IC_kwDOBm6k_c5Qn7a0,11729897,2022-12-14T14:53:53Z,2022-12-14T14:53:53Z,NONE,"I don't have much experience with Python; however, I wonder if this print statement needs `flush=True`? https://github.com/simonw/datasette/blob/fdf7c27b5438f02153c3a7f8ad1b320e4b29e4f4/datasette/cli.py#L621","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798, https://github.com/simonw/datasette/issues/1958#issuecomment-1352644274,https://api.github.com/repos/simonw/datasette/issues/1958,1352644274,IC_kwDOBm6k_c5Qn7ay,11729897,2022-12-14T14:19:24Z,2022-12-14T14:19:24Z,NONE,"Hmm, it appears after I kill the process with `Ctrl+c`: ![image](https://user-images.githubusercontent.com/11729897/207619736-a49b2e97-c5d0-4e54-ad3c-3c3c279927c7.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798, https://github.com/simonw/datasette/issues/1952#issuecomment-1350415644,https://api.github.com/repos/simonw/datasette/issues/1952,1350415644,IC_kwDOBm6k_c5QfbUc,9599,2022-12-14T05:22:59Z,2022-12-14T05:22:59Z,OWNER,"Non-memory named databases shouldn't show write actions, since those won't persist.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495716243, https://github.com/simonw/datasette/issues/1947#issuecomment-1350414961,https://api.github.com/repos/simonw/datasette/issues/1947,1350414961,IC_kwDOBm6k_c5QfbJx,9599,2022-12-14T05:22:00Z,2022-12-14T05:22:00Z,OWNER,I think the next big step for this feature is for me to actually use it to build a few things.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939, https://github.com/simonw/datasette/issues/1947#issuecomment-1350414402,https://api.github.com/repos/simonw/datasette/issues/1947,1350414402,IC_kwDOBm6k_c5QfbBC,9599,2022-12-14T05:21:07Z,2022-12-14T05:21:07Z,OWNER,"It would be neat not to show write permissions against immutable databases too - and not hard from a performance perspective since it doesn't involve hundreds more permission checks. That will need permissions to grow a flag for if they need a mutable database though, which is a bigger job.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939, https://github.com/simonw/datasette/issues/1947#issuecomment-1350413555,https://api.github.com/repos/simonw/datasette/issues/1947,1350413555,IC_kwDOBm6k_c5Qfazz,9599,2022-12-14T05:19:52Z,2022-12-14T05:19:52Z,OWNER,"Maybe I should have kept `_memory` listed for instances that are running with `--crossdb` enabled? Yeah I think I should.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939, https://github.com/simonw/datasette/issues/1947#issuecomment-1350409537,https://api.github.com/repos/simonw/datasette/issues/1947,1350409537,IC_kwDOBm6k_c5QfZ1B,9599,2022-12-14T05:14:16Z,2022-12-14T05:14:16Z,OWNER,"New interface now live at https://latest.datasette.io/-/create-token It shouldn't be showing `_memory` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939, https://github.com/simonw/datasette/issues/1947#issuecomment-1350402667,https://api.github.com/repos/simonw/datasette/issues/1947,1350402667,IC_kwDOBm6k_c5QfYJr,9599,2022-12-14T05:05:10Z,2022-12-14T05:05:10Z,OWNER,"Tests can go here: https://github.com/simonw/datasette/blob/d98a8effb10ce8fe04a03eae42baa8a9cb0ca3f7/tests/test_auth.py#L143-L160","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939, https://github.com/simonw/datasette/issues/1947#issuecomment-1350401651,https://api.github.com/repos/simonw/datasette/issues/1947,1350401651,IC_kwDOBm6k_c5QfX5z,9599,2022-12-14T05:03:59Z,2022-12-14T05:03:59Z,OWNER,I shipped a working interface. Could still do with some extra tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939, https://github.com/simonw/datasette/issues/1951#issuecomment-1350293098,https://api.github.com/repos/simonw/datasette/issues/1951,1350293098,IC_kwDOBm6k_c5Qe9Zq,9599,2022-12-14T02:43:44Z,2022-12-14T02:43:44Z,OWNER,Documentation for the new method: https://docs.datasette.io/en/latest/internals.html#create-token-actor-id-expires-after-none-restrict-all-none-restrict-database-none-restrict-resource-none,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495431932, https://github.com/simonw/datasette/issues/1951#issuecomment-1350231654,https://api.github.com/repos/simonw/datasette/issues/1951,1350231654,IC_kwDOBm6k_c5QeuZm,9599,2022-12-14T01:48:50Z,2022-12-14T01:48:57Z,OWNER,I like that the word `restrict` reflects the `_r` in the actor/token.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495431932, https://github.com/simonw/datasette/issues/1951#issuecomment-1350222701,https://api.github.com/repos/simonw/datasette/issues/1951,1350222701,IC_kwDOBm6k_c5QesNt,9599,2022-12-14T01:35:05Z,2022-12-14T01:35:22Z,OWNER,"Maybe this: ```python datasette.create_token(""root"", expires_after=3600, restrict_all=(""view-query"", ""view-table"")) token = datasette.create_token(""root"", expires_after=3600, restrict_database={ ""fixtures"": (""view-query"",) }) token = datasette.create_token(""root"", expires_after=3600, restrict_resource={ ""fixtures"": { ""facetable"": (""insert-row"", ""update-row"") } }) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495431932, https://github.com/simonw/datasette/issues/1951#issuecomment-1350220579,https://api.github.com/repos/simonw/datasette/issues/1951,1350220579,IC_kwDOBm6k_c5Qersj,9599,2022-12-14T01:31:38Z,2022-12-14T01:31:38Z,OWNER,"The problem with `all=(..)` is it feels misleading - it's actually restricting the permissions made available to the token. Likewise, `databases=` being a dict of restricted permissions isn't completely obvious. And the nested `tables=` dictionary feels a bit odd too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495431932, https://github.com/simonw/datasette/issues/1951#issuecomment-1350218177,https://api.github.com/repos/simonw/datasette/issues/1951,1350218177,IC_kwDOBm6k_c5QerHB,9599,2022-12-14T01:27:45Z,2022-12-14T01:30:41Z,OWNER,"Some sketches: ```python # Token for root user token = datasette.create_token(""root"") # Expiring in an hour token = datasette.create_token(""root"", expires_after=3600) ``` More complicated is when you want to restrict to specific permissions: ```python # Limited to view-query and view-table token = datasette.create_token(""root"", expires_after=3600, all=(""view-query"", ""view-table"")) # I'm not sure about that all= name # Limits within a specific database: token = datasette.create_token(""root"", expires_after=3600, databases={ ""fixtures"": (""view-query"",) }) # And specific tables: token = datasette.create_token(""root"", expires_after=3600, tables={ ""fixtures"": { ""facetable"": (""insert-row"", ""update-row"") } }) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495431932, https://github.com/simonw/datasette/issues/1951#issuecomment-1350217380,https://api.github.com/repos/simonw/datasette/issues/1951,1350217380,IC_kwDOBm6k_c5Qeq6k,9599,2022-12-14T01:26:22Z,2022-12-14T01:26:22Z,OWNER,"It's going to look very similar to the CLI tool, at least in terms of capabilities: ``` Usage: datasette create-token [OPTIONS] ID Create a signed API token for the specified actor ID Example: datasette create-token root --secret mysecret To allow only ""view-database-download"" for all databases: datasette create-token root --secret mysecret \ --all view-database-download To allow ""create-table"" against a specific database: datasette create-token root --secret mysecret \ --database mydb create-table To allow ""insert-row"" against a specific table: datasette create-token root --secret myscret \ --resource mydb mytable insert-row Restricted actions can be specified multiple times using multiple --all, --database, and --resource options. Add --debug to see a decoded version of the token. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495431932,