issue_comments
10,495 rows sorted by updated_at descending
This data as json, CSV (advanced)
issue >30
- Show column metadata plus links for foreign keys on arbitrary query results 51
- Redesign default .json format 48
- Rethink how .ext formats (v.s. ?_format=) works before 1.0 48
- Upgrade to CodeMirror 6, add SQL autocomplete 48
- JavaScript plugin hooks mechanism similar to pluggy 47
- Updated Dockerfile with SpatiaLite version 5.0 45
- Complete refactor of TableView and table.html template 45
- Port Datasette to ASGI 42
- Authentication (and permissions) as a core concept 40
- Deploy a live instance of demos/apache-proxy 34
- await datasette.client.get(path) mechanism for executing internal requests 33
- Maintain an in-memory SQLite table of connected databases and their tables 32
- Research: demonstrate if parallel SQL queries are worthwhile 32
- Ability to sort (and paginate) by column 31
- Default API token authentication mechanism 30
- Port as many tests as possible to async def tests against ds_client 29
- link_or_copy_directory() error - Invalid cross-device link 28
- Export to CSV 27
- base_url configuration setting 27
- Documentation with recommendations on running Datasette in production without using Docker 27
- Optimize all those calls to index_list and foreign_key_list 27
- Support cross-database joins 26
- Ability for a canned query to write to the database 26
- table.transform() method for advanced alter table 26
- New pattern for views that return either JSON or HTML, available for plugins 26
- Proof of concept for Datasette on AWS Lambda with EFS 25
- WIP: Add Gmail takeout mbox import 25
- Redesign register_output_renderer callback 24
- Make it easier to insert geometries, with documentation and maybe code 24
- API explorer tool 24
- …
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
1356487139 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356487139 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q2lnj | simonw 9599 | 2022-12-17T22:16:52Z | 2022-12-17T22:16:52Z | OWNER | I'm trying this fix again, after a bunch of work on the test suite in: - #1959 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356481595 | https://github.com/simonw/datasette/issues/1843#issuecomment-1356481595 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5Q2kQ7 | simonw 9599 | 2022-12-17T22:02:40Z | 2022-12-17T22:02:40Z | OWNER | This is interesting:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1356480256 | https://github.com/simonw/datasette/issues/1843#issuecomment-1356480256 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5Q2j8A | simonw 9599 | 2022-12-17T22:01:13Z | 2022-12-17T22:01:13Z | OWNER | The refactor in the following issue did NOT prevent this error from occurring when I try to run the full
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1356478792 | https://github.com/simonw/datasette/issues/1959#issuecomment-1356478792 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5Q2jlI | simonw 9599 | 2022-12-17T21:49:36Z | 2022-12-17T21:49:36Z | OWNER | Made a really good start on this in the just-merged PR: - #1960 The follow-up work will happen in: - #1962 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1356478575 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356478575 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q2jhv | simonw 9599 | 2022-12-17T21:47:48Z | 2022-12-17T21:47:48Z | OWNER | Stick a twig in it, this will do for the moment. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354062939 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354062939 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtVxb | codecov[bot] 22429695 | 2022-12-16T01:48:06Z | 2022-12-17T21:40:43Z | NONE | Codecov ReportBase: 92.19% // Head: 92.05% // Decreases project coverage by
Additional details and impacted files```diff @@ Coverage Diff @@ ## main #1960 +/- ## ========================================== - Coverage 92.19% 92.05% -0.14% ========================================== Files 38 38 Lines 5521 5527 +6 ========================================== - Hits 5090 5088 -2 - Misses 431 439 +8 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.13% <100.00%> (-0.57%)` | :arrow_down: | | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.83% <100.00%> (+0.24%)` | :arrow_up: | | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: | | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.57% <0.00%> (-0.61%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison):umbrella: View full report at Codecov. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356476886 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356476886 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q2jHW | simonw 9599 | 2022-12-17T21:37:05Z | 2022-12-17T21:37:05Z | OWNER | I think this test may be to blame: It's over-riding |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356476583 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356476583 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q2jCn | simonw 9599 | 2022-12-17T21:34:51Z | 2022-12-17T21:34:51Z | OWNER | These are all the places that tests touch test_permissions.py 62: padlock_client.ds._metadata_local["allow"] = allow 77: del padlock_client.ds._metadata_local["allow"] 522: cascade_app_client.ds._metadata_local = updated_metadata 533: cascade_app_client.ds._metadata_local = previous_metadata 549: previous_metadata = cascade_app_client.ds._metadata_local 551: cascade_app_client.ds._metadata_local = metadata 566: cascade_app_client.ds._metadata_local = previous_metadata 842: perms_ds._metadata_local = updated_metadata 849: perms_ds._metadata_local = previous_metadata test_html.py 1114: orig = ds_client.ds._metadata_local 1115: ds_client.ds._metadata_local = metadata 1123: ds_client.ds._metadata_local = orig test_plugins.py 1034: ds_client.ds._metadata_local = { ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356056018 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356056018 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q08XS | simonw 9599 | 2022-12-17T05:33:26Z | 2022-12-17T05:33:26Z | OWNER | I think I've found the problem. The failing test is this one:
That's this example: Why is it expected to take 9 pages and not 5, when the definition of that view is this:
So this test is showing that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356038242 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356038242 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q04Bi | simonw 9599 | 2022-12-17T04:56:22Z | 2022-12-17T04:57:04Z | OWNER | May have spotted the problem with that
No that's not it either - |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355823260 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355823260 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q0Dic | simonw 9599 | 2022-12-16T23:36:07Z | 2022-12-16T23:36:07Z | OWNER | I ran ``` assert expected_rows == len(fetched)
On a hunch, I checked the current settings:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355691595 | https://github.com/simonw/datasette/issues/1962#issuecomment-1355691595 | https://api.github.com/repos/simonw/datasette/issues/1962 | IC_kwDOBm6k_c5QzjZL | simonw 9599 | 2022-12-16T21:53:45Z | 2022-12-16T21:55:29Z | OWNER | Also need an alternative mechanism to this convenience for getting CSRF tokens before a POST: One option would be adding that mechanism to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Alternative, async-friendly pattern for `make_app_client()` and similar - fully retire `TestClient` 1500636982 | |
1355685828 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355685828 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Qzh_E | simonw 9599 | 2022-12-16T21:50:01Z | 2022-12-16T21:50:01Z | OWNER | Looks like that
So I'm going to remove the mark. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355478743 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355478743 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyvbX | simonw 9599 | 2022-12-16T19:27:12Z | 2022-12-16T19:27:12Z | OWNER | Bad news: they're definitely caused by tests that are subtly affected by other tests. This passes without errors:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355475671 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355475671 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyurX | simonw 9599 | 2022-12-16T19:26:08Z | 2022-12-16T19:26:08Z | OWNER | Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355471341 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355471341 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Qytnt | simonw 9599 | 2022-12-16T19:23:33Z | 2022-12-16T19:23:33Z | OWNER | Trying this instead:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355445710 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355445710 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QynXO | simonw 9599 | 2022-12-16T19:06:49Z | 2022-12-16T19:09:35Z | OWNER | This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop. https://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though. https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/ |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355325426 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355325426 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyJ_y | simonw 9599 | 2022-12-16T18:00:40Z | 2022-12-16T18:00:40Z | OWNER | Many of the failing tests pass on my laptop but fail in CI. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355319541 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355319541 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyIj1 | simonw 9599 | 2022-12-16T17:58:24Z | 2022-12-16T17:58:46Z | OWNER |
It did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355317782 | https://github.com/simonw/datasette/issues/1959#issuecomment-1355317782 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QyIIW | simonw 9599 | 2022-12-16T17:57:25Z | 2022-12-16T17:57:25Z | OWNER | Opened a follow-up issue here: - #1962 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1355317369 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355317369 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyIB5 | simonw 9599 | 2022-12-16T17:57:14Z | 2022-12-16T17:57:14Z | OWNER | Opened a follow-up issue here: - #1962 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355313058 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355313058 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyG-i | simonw 9599 | 2022-12-16T17:53:17Z | 2022-12-16T17:53:17Z | OWNER | Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626 ```
I tried adding |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355300217 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355300217 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyD15 | simonw 9599 | 2022-12-16T17:44:55Z | 2022-12-16T17:44:55Z | OWNER | That's enough for this round. I'll get the tests passing and land this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354269873 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354269873 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QuISx | simonw 9599 | 2022-12-16T06:11:43Z | 2022-12-16T06:11:43Z | OWNER | This is quite fast:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354192168 | https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354192168 | https://api.github.com/repos/simonw/sqlite-utils/issues/235 | IC_kwDOCGYnMM5Qt1Uo | hydrosquall 9020979 | 2022-12-16T04:35:30Z | 2022-12-16T04:35:38Z | NONE | A related historical problem: https://github.com/tekartik/sqflite/issues/525#issuecomment-714500720 I wonder if the version of Sqlite or Python for Intel chip have defensive mode disabled by default, whereas M1 chips versions have it enabled. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified 810618495 | |
1354160286 | https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354160286 | https://api.github.com/repos/simonw/sqlite-utils/issues/235 | IC_kwDOCGYnMM5Qttie | hydrosquall 9020979 | 2022-12-16T03:51:19Z | 2022-12-16T03:52:13Z | NONE | Hi @ryascott, thanks for sharing this! How did you upgrade your sqlite3 version? I'm running into this issue (also on an m1) with
Unfortunately, For me, the trigger is trying to use the
Some stackoverflow searching suggests that brew installing sqlite may fix it ( https://stackoverflow.com/questions/26345972/how-do-i-upgrade-the-sqlite-version-used-by-pythons-sqlite3-module-on-mac ), but I don't want to risk breaking the version of sqlite used by some other system, I'd only like to upgrade sqlite3 inside my current virtual environment. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified 810618495 | |
1354148139 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354148139 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Qtqkr | simonw 9599 | 2022-12-16T03:32:52Z | 2022-12-16T03:32:52Z | OWNER | Got that done to: 68 passed in 14.92s By implementing my own global variable - since |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354072344 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354072344 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtYEY | simonw 9599 | 2022-12-16T02:00:07Z | 2022-12-16T02:00:07Z | OWNER | It did NOT speed it up: 68 passed in 26.26s |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354061440 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354061440 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtVaA | simonw 9599 | 2022-12-16T01:45:38Z | 2022-12-16T01:45:38Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | ||
1354053151 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354053151 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtTYf | simonw 9599 | 2022-12-16T01:33:22Z | 2022-12-16T01:33:22Z | OWNER | The thing with The problem is that the tests written against But the new This results in a datasette instance that DOES have a My initial solution attempt was a huge hack - I decided that if you pass This is weird and surprising (
So I'm going to undo that hack and teach the fixture to do this instead:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354046627 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354046627 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtRyj | simonw 9599 | 2022-12-16T01:23:21Z | 2022-12-16T01:23:21Z | OWNER | This does seem to help: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 1306c407..af9c7696 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = { } -@pytest.fixture(scope="session") -def event_loop(): - return asyncio.get_event_loop() - - -@pytest_asyncio.fixture(scope="session") +@pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette from .fixtures import METADATA, PLUGINS_DIR @@ -53,10 +48,11 @@ async def ds_client(): db = ds.add_memory_database("fixtures")
Need to get |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354036967 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354036967 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtPbn | simonw 9599 | 2022-12-16T01:10:12Z | 2022-12-16T01:10:12Z | OWNER | If it does turn out that I can't use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354025319 | https://github.com/simonw/datasette/issues/1958#issuecomment-1354025319 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtMln | davidhaley 11729897 | 2022-12-16T00:59:12Z | 2022-12-16T00:59:12Z | NONE | Awesome. Thank you @simonw. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1354023960 | https://github.com/simonw/datasette/issues/1958#issuecomment-1354023960 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtMQY | simonw 9599 | 2022-12-16T00:58:14Z | 2022-12-16T00:58:19Z | OWNER | This is in the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1354019543 | https://github.com/simonw/datasette/issues/1958#issuecomment-1354019543 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtLLX | simonw 9599 | 2022-12-16T00:54:11Z | 2022-12-16T00:54:11Z | OWNER | To test the fix, I did a fresh checkout of RUN apt-get update && \ apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg/info/* RUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \ find /usr/local/lib -name 'pycache' | xargs rm -r && \ rm -rf /root/.cache/pip EXPOSE 8001
CMD ["datasette"]
Then I pushed a changed to branch and changed the line to:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1354008688 | https://github.com/simonw/datasette/issues/1958#issuecomment-1354008688 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtIhw | simonw 9599 | 2022-12-16T00:45:40Z | 2022-12-16T00:45:40Z | OWNER | The fix may just be to switch to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1353977605 | https://github.com/simonw/datasette/issues/1958#issuecomment-1353977605 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtA8F | simonw 9599 | 2022-12-16T00:38:23Z | 2022-12-16T00:38:23Z | OWNER | Confirmed, I just got the same result: ``` - % docker run datasetteproject/datasette pip install datasette-upload-csvs ~ % docker commit $(docker ps -lq) datasette-with-plugins sha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f ~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0 INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ^CINFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [1]
http://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1353812913 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353812913 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsYux | simonw 9599 | 2022-12-15T22:48:54Z | 2022-12-15T22:48:54Z | OWNER | This is all very broken: ``` % pytest -x --pdb ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items tests/test_package.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_cli_serve_get.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_black.py . [ 0%] tests/test_api.py E
fixturedef = <FixtureDef argname='event_loop' scope='session' baseid='tests'>, request = <SubRequest 'event_loop' for \<Function test_homepage>>
/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pytest_asyncio/plugin.py:377: /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: in event_loop return asyncio.get_event_loop() self = <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x111e34670>
/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py:656: RuntimeError
=================================================================================== warnings summary ==================================================================================== tests/test_cli.py::test_inspect_cli_writes_to_file tests/test_cli.py::test_inspect_cli /Users/simon/Dropbox/Development/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop loop = asyncio.get_event_loop() tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error tests/test_cli_serve_get.py::test_serve_with_get /Users/simon/Dropbox/Development/datasette/datasette/cli.py:596: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(ds.invoke_startup()) tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error tests/test_cli_serve_get.py::test_serve_with_get /Users/simon/Dropbox/Development/datasette/datasette/cli.py:599: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(check_databases(ds)) tests/test_api.py::test_homepage /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: DeprecationWarning: There is no current event loop return asyncio.get_event_loop() -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================================ short test summary info ================================================================================ ERROR tests/test_api.py::test_homepage - RuntimeError: There is no current event loop in thread 'MainThread'. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Quitting debugger !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ======================================================================== 7 passed, 7 warnings, 1 error in 19.15s ======================================================================== (datasette) datasette % ``` The problem looks to be caused by this: Which I found necessary in order to have Can I work around this, or is |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353805839 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353805839 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsXAP | simonw 9599 | 2022-12-15T22:38:37Z | 2022-12-15T22:38:37Z | OWNER | I'm going to make |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353765125 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353765125 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsNEF | simonw 9599 | 2022-12-15T22:00:04Z | 2022-12-15T22:00:04Z | OWNER | I'm going to punt on that for the moment and continue to use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353763837 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353763837 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsMv9 | simonw 9599 | 2022-12-15T21:59:05Z | 2022-12-15T21:59:05Z | OWNER | Here's an annoying error: ```
tests/test_auth.py:88: self = <datasette.app.DatasetteClient object at 0x1033cc4f0>, path = '/-/logout' kwargs = {'cookies': {'ds_actor': 'eyJhIjp7ImlkIjoidGVzdCJ9fQ.fuFCTJG5XE-RNnUM7dcnXx9sPvE'}, 'csrftoken_from': True}, client = <httpx.AsyncClient object at 0x10347a9b0>
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353749401 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353749401 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsJOZ | simonw 9599 | 2022-12-15T21:47:27Z | 2022-12-15T21:47:27Z | OWNER | I'm using this new mark:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353747370 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353747370 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsIuq | simonw 9599 | 2022-12-15T21:45:14Z | 2022-12-15T21:45:14Z | OWNER | I'm going to do this in a PR. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353738075 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353738075 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsGdb | simonw 9599 | 2022-12-15T21:35:56Z | 2022-12-15T21:35:56Z | OWNER | I built that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353728682 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353728682 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsEKq | simonw 9599 | 2022-12-15T21:28:35Z | 2022-12-15T21:28:35Z | OWNER | Got this error trying to have two tests use the same Adding this to
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353721442 | https://github.com/simonw/datasette/issues/1619#issuecomment-1353721442 | https://api.github.com/repos/simonw/datasette/issues/1619 | IC_kwDOBm6k_c5QsCZi | noslouch 2090382 | 2022-12-15T21:20:53Z | 2022-12-15T21:20:53Z | NONE | i'm also getting bit by this. I'm trying to set up an nginx reverse proxy in front of multiple datasette backends. When I run it locally or behind the proxy, I see the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
JSON link on row page is 404 if base_url setting is used 1121583414 | |
1353721091 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsCUD | simonw 9599 | 2022-12-15T21:20:32Z | 2022-12-15T21:20:32Z | OWNER | Rather than tediously rewriting every single test to the new shape I'm going to try a wrapper for that HTTPX response that transforms it into an imitation of the one returned by the existing |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353720559 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353720559 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsCLv | simonw 9599 | 2022-12-15T21:19:56Z | 2022-12-15T21:19:56Z | OWNER | Here's a port of the first ```diff diff --git a/datasette/app.py b/datasette/app.py index f3cb8876..b770b469 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -281,7 +281,7 @@ class Datasette: raise self.crossdb = crossdb self.nolock = nolock - if memory or crossdb or not self.files: + if memory or crossdb or (not self.files and memory is not False): self.add_database( Database(self, is_mutable=False, is_memory=True), name="_memory" ) diff --git a/pytest.ini b/pytest.ini index 559e518c..0bcb0d1e 100644 --- a/pytest.ini +++ b/pytest.ini @@ -8,4 +8,5 @@ filterwarnings= ignore:.current_task.:PendingDeprecationWarning markers = serial: tests to avoid using with pytest-xdist + ds_client: tests using the ds_client fixture asyncio_mode = strict diff --git a/tests/conftest.py b/tests/conftest.py index cd735e12..648423ba 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -2,6 +2,7 @@ import httpx import os import pathlib import pytest +import pytest_asyncio import re import subprocess import tempfile @@ -23,6 +24,22 @@ UNDOCUMENTED_PERMISSIONS = { } +@pytest_asyncio.fixture +async def ds_client(): + from datasette.app import Datasette + from .fixtures import METADATA, PLUGINS_DIR + ds = Datasette(memory=False, metadata=METADATA, plugins_dir=PLUGINS_DIR) + from .fixtures import TABLES, TABLE_PARAMETERIZED_SQL + db = ds.add_memory_database("fixtures") + def prepare(conn): + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) + await db.execute_write_fn(prepare) + return ds.client + + def pytest_report_header(config): return "SQLite: {}".format( sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0] diff --git a/tests/test_api.py b/tests/test_api.py index 5f2a6ea6..ddf4219c 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -23,12 +23,15 @@ import sys import urllib -def test_homepage(app_client): - response = app_client.get("/.json") - assert response.status == 200 +@pytest.mark.ds_client +@pytest.mark.asyncio +async def test_homepage(ds_client): + response = await ds_client.get("/.json") + assert response.status_code == 200 assert "application/json; charset=utf-8" == response.headers["content-type"] - assert response.json.keys() == {"fixtures": 0}.keys() - d = response.json["fixtures"] + data = response.json() + assert data.keys() == {"fixtures": 0}.keys() + d = data["fixtures"] assert d["name"] == "fixtures" assert d["tables_count"] == 24 assert len(d["tables_and_views_truncated"]) == 5 @@ -36,7 +39,7 @@ def test_homepage(app_client): # 4 hidden FTS tables + no_primary_key (hidden in metadata) assert d["hidden_tables_count"] == 6 # 201 in no_primary_key, plus 6 in other hidden tables: - assert d["hidden_table_rows_sum"] == 207, response.json + assert d["hidden_table_rows_sum"] == 207, data assert d["views_count"] == 4 ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353707828 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353707828 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5Qr_E0 | simonw 9599 | 2022-12-15T21:06:29Z | 2022-12-15T21:06:29Z | OWNER | Previous, abandoned attempt at this work (for #1843): ```diff diff --git a/datasette/app.py b/datasette/app.py index 7e682498..cf35c3a2 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -228,7 +228,7 @@ class Datasette: template_dir=None, plugins_dir=None, static_mounts=None, - memory=False, + memory=None, settings=None, secret=None, version_note=None, @@ -238,6 +238,7 @@ class Datasette: nolock=False, ): self._startup_invoked = False + self._extra_on_startup = [] assert config_dir is None or isinstance( config_dir, Path ), "config_dir= should be a pathlib.Path" @@ -278,7 +279,7 @@ class Datasette: raise self.crossdb = crossdb self.nolock = nolock - if memory or crossdb or not self.files: + if memory or crossdb or (not self.files and memory is not False): self.add_database( Database(self, is_mutable=False, is_memory=True), name="_memory" ) @@ -391,6 +392,9 @@ class Datasette: self._root_token = secrets.token_hex(32) self.client = DatasetteClient(self)
diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py index 56690251..986755cb 100644 --- a/datasette/utils/asgi.py +++ b/datasette/utils/asgi.py @@ -423,9 +423,9 @@ class AsgiFileDownload: class AsgiRunOnFirstRequest: - def init(self, asgi, on_startup): + def init(self, app, on_startup): assert isinstance(on_startup, list) - self.asgi = asgi + self.app = app self.on_startup = on_startup self._started = False @@ -434,4 +434,4 @@ class AsgiRunOnFirstRequest: self._started = True for hook in self.on_startup: await hook() - return await self.asgi(scope, receive, send) + return await self.app(scope, receive, send) diff --git a/tests/conftest.py b/tests/conftest.py index cd735e12..d1301943 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -23,6 +23,15 @@ UNDOCUMENTED_PERMISSIONS = { } +# @pytest.fixture(autouse=True) +# def log_name_of_test_before_test(request): +# # To help identify tests that are hanging +# name = str(request.node) +# with open("/tmp/test.log", "a") as f: +# f.write(name + "\n") +# yield + + def pytest_report_header(config): return "SQLite: {}".format( sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0] diff --git a/tests/fixtures.py b/tests/fixtures.py index a6700239..18d3f1b7 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -101,6 +101,19 @@ EXPECTED_PLUGINS = [ ] +def _populate_connection(conn): + # Drop any tables and views that exist + to_drop = conn.execute( + "SELECT name, type FROM sqlite_master where type in ('table', 'view')" + ).fetchall() + for name, type in to_drop: + conn.execute(f"DROP {type} IF EXISTS [{name}]") + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) + + @contextlib.contextmanager def make_app_client( sql_time_limit_ms=None, @@ -117,45 +130,22 @@ def make_app_client( metadata=None, crossdb=False, ): - with tempfile.TemporaryDirectory() as tmpdir: - filepath = os.path.join(tmpdir, filename) - if is_immutable: - files = [] - immutables = [filepath] - else: - files = [filepath] - immutables = [] - conn = sqlite3.connect(filepath) - conn.executescript(TABLES) - for sql, params in TABLE_PARAMETERIZED_SQL: - with conn: - conn.execute(sql, params) - # Close the connection to avoid "too many open files" errors - conn.close() - if extra_databases is not None: - for extra_filename, extra_sql in extra_databases.items(): - extra_filepath = os.path.join(tmpdir, extra_filename) - c2 = sqlite3.connect(extra_filepath) - c2.executescript(extra_sql) - c2.close() - # Insert at start to help test /-/databases ordering: - files.insert(0, extra_filepath) - os.chdir(os.path.dirname(filepath)) - settings = settings or {} - for key, value in { - "default_page_size": 50, - "max_returned_rows": max_returned_rows or 100, - "sql_time_limit_ms": sql_time_limit_ms or 200, - # Default is 3 but this results in "too many open files" - # errors when running the full test suite: - "num_sql_threads": 1, - }.items(): - if key not in settings: - settings[key] = value + settings = settings or {} + for key, value in { + "default_page_size": 50, + "max_returned_rows": max_returned_rows or 100, + "sql_time_limit_ms": sql_time_limit_ms or 200, + # Default is 3 but this results in "too many open files" + # errors when running the full test suite: + "num_sql_threads": 1, + }.items(): + if key not in settings: + settings[key] = value + # We can use an in-memory database, but only if we're not doing anything + # with is_immutable or extra_databases and filename is the default + if not is_immutable and not extra_databases and filename == "fixtures.db": ds = Datasette( - files, - immutables=immutables, - memory=memory, + memory=memory or False, cors=cors, metadata=metadata or METADATA, plugins_dir=PLUGINS_DIR, @@ -165,12 +155,57 @@ def make_app_client( template_dir=template_dir, crossdb=crossdb, ) + db = ds.add_memory_database("fixtures") + + async def populate_fixtures(): + print("Here we go... populating fixtures") + await db.execute_write_fn(_populate_connection) + + ds._add_on_startup(populate_fixtures) yield TestClient(ds) - # Close as many database connections as possible - # to try and avoid too many open files error - for db in ds.databases.values(): - if not db.is_memory: - db.close() + else: + with tempfile.TemporaryDirectory() as tmpdir: + filepath = os.path.join(tmpdir, filename) + if is_immutable: + files = [] + immutables = [filepath] + else: + files = [filepath] + immutables = [] + + conn = sqlite3.connect(filepath) + _populate_connection(conn) + # Close the connection to reduce "too many open files" errors + conn.close() + + if extra_databases is not None: + for extra_filename, extra_sql in extra_databases.items(): + extra_filepath = os.path.join(tmpdir, extra_filename) + c2 = sqlite3.connect(extra_filepath) + c2.executescript(extra_sql) + c2.close() + # Insert at start to help test /-/databases ordering: + files.insert(0, extra_filepath) + os.chdir(os.path.dirname(filepath)) + ds = Datasette( + files, + immutables=immutables, + memory=memory, + cors=cors, + metadata=metadata or METADATA, + plugins_dir=PLUGINS_DIR, + settings=settings, + inspect_data=inspect_data, + static_mounts=static_mounts, + template_dir=template_dir, + crossdb=crossdb, + ) + yield TestClient(ds) + # Close as many database connections as possible + # to try and avoid too many open files error + for db in ds.databases.values(): + if not db.is_memory: + db.close() @pytest.fixture(scope="session") diff --git a/tests/test_cli.py b/tests/test_cli.py index d3e015fa..d9e4e457 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,5 +1,6 @@ from .fixtures import ( app_client, + app_client_with_cors, make_app_client, TestClient as _TestClient, EXPECTED_PLUGINS, @@ -38,7 +39,7 @@ def test_inspect_cli(app_client): assert expected_count == database["tables"][table_name]["count"] -def test_inspect_cli_writes_to_file(app_client): +def test_inspect_cli_writes_to_file(app_client_with_cors): runner = CliRunner() result = runner.invoke( cli, ["inspect", "fixtures.db", "--inspect-file", "foo.json"] ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353705072 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353705072 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5Qr-Zw | simonw 9599 | 2022-12-15T21:04:07Z | 2022-12-15T21:04:07Z | OWNER | I'm going to start by getting every test that uses the raw |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353701674 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353701674 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr9kq | simonw 9599 | 2022-12-15T21:00:51Z | 2022-12-15T21:00:51Z | OWNER | OK, I've broken the test suite here. I'm going to revert these two commits:
Then I'll do a bunch of work making the test suite more robust before I try this again. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353694582 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353694582 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr712 | simonw 9599 | 2022-12-15T20:52:46Z | 2022-12-15T20:52:46Z | OWNER | Just noticed this: https://github.com/simonw/datasette/actions/runs/3706504228/jobs/6281796135 This suggests that the regular tests passed in CI fine, but the non-serial ones failed. I'm going to try running everything using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353690591 | https://github.com/simonw/datasette/issues/1843#issuecomment-1353690591 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5Qr63f | simonw 9599 | 2022-12-15T20:49:05Z | 2022-12-15T20:49:05Z | OWNER | I have a nasty feeling the cleaner solution for this would involve porting my entire test suite from I've been using that latter pattern for new tests (and plugin tests) for quite a while now, but I never took on the job of refactoring all of the old ones. A search for No idea if it will have much impact on the "Too many open files" errors though. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1353683238 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353683238 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr5Em | simonw 9599 | 2022-12-15T20:42:18Z | 2022-12-15T20:42:18Z | OWNER | Possibly related issue: - https://github.com/pytest-dev/pytest-xdist/issues/60 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353680261 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353680261 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr4WF | simonw 9599 | 2022-12-15T20:39:19Z | 2022-12-15T20:39:19Z | OWNER | When I hit |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353522652 | https://github.com/simonw/datasette/issues/1843#issuecomment-1353522652 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5QrR3c | simonw 9599 | 2022-12-15T18:21:27Z | 2022-12-15T18:21:27Z | OWNER | I'll still use on-disk test databases for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1353522211 | https://github.com/simonw/datasette/issues/1843#issuecomment-1353522211 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5QrRwj | simonw 9599 | 2022-12-15T18:21:02Z | 2022-12-15T18:21:02Z | OWNER | When I initially built this test suite Datasette didn't have the I'm going to see if I can switch to that for the majority of Datasette's tests. Might find that doing so both fixes this "too many open files" issue AND gives me a significant speed improvement to the test site too! Hopefully I can do most of the work on that in this big ugly function: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1353520615 | https://github.com/simonw/datasette/issues/1843#issuecomment-1353520615 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5QrRXn | simonw 9599 | 2022-12-15T18:19:25Z | 2022-12-15T18:19:25Z | OWNER | I've been seeing this error again:
It doesn't happen in CI, and it turns out that's because CI runs I've been working around the error on my laptop using This is also getting in my way when I try to debug other issues, like this one:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1353516572 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353516572 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrQYc | simonw 9599 | 2022-12-15T18:15:28Z | 2022-12-15T18:15:28Z | OWNER | I added So this time it was hanging at So it's clearly not the individual tests themselves that are the problem - something about running the entire test suite in one go is incompatible with this change for some reason. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353512099 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353512099 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrPSj | simonw 9599 | 2022-12-15T18:11:27Z | 2022-12-15T18:11:27Z | OWNER | This is surprising! The logs suggest that the test suite hung running this test here: I find that very hard to believe. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353509776 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353509776 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrOuQ | simonw 9599 | 2022-12-15T18:09:26Z | 2022-12-15T18:09:26Z | OWNER | I added this to
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353473571 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353473571 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrF4j | simonw 9599 | 2022-12-15T17:43:28Z | 2022-12-15T17:43:48Z | OWNER | Running:
On may laptop to see if I can replicate. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353473086 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353473086 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrFw- | simonw 9599 | 2022-12-15T17:43:08Z | 2022-12-15T17:43:08Z | OWNER | It looks like that fix almost works... except it seems to push the tests into an infinite loop or similar? They're not finishing their runs from what I can see. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353448095 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353448095 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qq_qf | simonw 9599 | 2022-12-15T17:25:05Z | 2022-12-15T17:25:05Z | OWNER | So actually that I'm inclined to ditch |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353443718 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353443718 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qq-mG | simonw 9599 | 2022-12-15T17:23:12Z | 2022-12-15T17:23:55Z | OWNER | That may not be the best fix here. It turns out this pattern:
I wrote about that previously in this TIL: https://til.simonwillison.net/asgi/lifespan-test-httpx |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353423584 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353423584 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qq5rg | simonw 9599 | 2022-12-15T17:13:18Z | 2022-12-15T17:22:59Z | OWNER | Wow, just spotted this in the code - it turns out I solved this problem a different (and better) way long before i introduced |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1352674924 | https://github.com/simonw/datasette/issues/1955#issuecomment-1352674924 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QoC5s | simonw 9599 | 2022-12-15T07:46:36Z | 2022-12-15T07:46:36Z | OWNER | It's possible the fix for this might be for the first incoming HTTP request to trigger This would be a much more elegant fix, I could remove those multiple |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1352644281 | https://github.com/simonw/datasette/issues/1958#issuecomment-1352644281 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5Qn7a5 | simonw 9599 | 2022-12-15T07:08:14Z | 2022-12-15T07:08:14Z | OWNER | Thanks for the details write-up! This looks like a bug in Datasette itself when run with Docker. Moving this issue there. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1352643333 | https://github.com/simonw/datasette/issues/1955#issuecomment-1352643333 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qn7MF | simonw 9599 | 2022-12-15T07:07:29Z | 2022-12-15T07:07:29Z | OWNER | Datasette 0.63 is the release that broke this, thanks to this issue: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1352643049 | https://github.com/simonw/datasette/issues/1955#issuecomment-1352643049 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qn7Hp | simonw 9599 | 2022-12-15T07:07:10Z | 2022-12-15T07:07:10Z | OWNER | This is definitely a regression: Datasette is meant to work in those environments, and I didn't think to test them when I added the Coincidentally I actually built a plugin for running Datasette with Gunicorn just a couple of months ago: https://datasette.io/plugins/datasette-gunicorn And I just tested and it has the same bug you describe here! Filed: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1352459146 | https://github.com/simonw/datasette/issues/1953#issuecomment-1352459146 | https://api.github.com/repos/simonw/datasette/issues/1953 | IC_kwDOBm6k_c5QnOOK | simonw 9599 | 2022-12-15T02:02:15Z | 2022-12-15T02:02:15Z | OWNER | ``` The third Datasette 1.0 alpha release adds upsert support to the JSON API, plus the ability to specify finely grained permissions when creating an API token.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release notes for Datasette 1.0a2 1495821607 | |
1352411327 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352411327 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5QnCi_ | simonw 9599 | 2022-12-15T00:46:27Z | 2022-12-15T00:46:27Z | OWNER | I got this far: ```diff diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index 8b7e83e3..31d41e00 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -54,7 +54,17 @@ def handle_exception(datasette, request, exception): headers = {} if datasette.cors: add_cors_headers(headers) - if request.path.split("?")[0].endswith(".json"): + # Return JSON error under certain conditions + should_return_json = ( + # URL ends in .json + request.path.split("?")[0].endswith(".json") + or + # Hints from incoming request headers + request.headers.get("content-type") == "application/json" + or "application/json" in request.headers.get("accept", "") + ) + breakpoint() + if should_return_json: return Response.json(info, status=status, headers=headers) else: template = datasette.jinja_env.select_template(templates) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index f27d143f..982543a6 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1140,6 +1140,38 @@ async def test_create_table_permissions( assert data["errors"] == expected_errors +@pytest.mark.asyncio +@pytest.mark.parametrize( + "headers,expect_json", + ( + ({}, False), + ({"Accept": "text/html"}, True), + ({"Accept": "application/json"}, True), + ({"Content-Type": "application/json"}, True), + ({"Accept": "application/json, text/plain, /"}, True), + ({"Content-Type": "application/json"}, True), + ({"accept": "application/json, text/plain, /"}, True), + ({"content-type": "application/json"}, True), + ), +) +async def test_permission_errors_html_and_json(ds_write, headers, expect_json): + request_headers = {"Authorization": "Bearer bad_token"} + request_headers.update(headers) + response = await ds_write.client.post( + "/data/-/create", + json={}, + headers=request_headers, + ) + assert response.status_code == 403 + if expect_json: + data = response.json() + assert data["ok"] is False + assert data["errors"] == ["Permission denied"] + else: + assert response.headers["Content-Type"] == "text/html; charset=utf-8" + assert "Permission denied" in response.text + + @pytest.mark.asyncio @pytest.mark.parametrize( "input,expected_rows_after", ``` Then decided I would punt this until the next milestone. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352410078 | https://github.com/simonw/datasette/issues/1953#issuecomment-1352410078 | https://api.github.com/repos/simonw/datasette/issues/1953 | IC_kwDOBm6k_c5QnCPe | simonw 9599 | 2022-12-15T00:44:56Z | 2022-12-15T00:44:56Z | OWNER | Highlights:
Plus some smaller things. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release notes for Datasette 1.0a2 1495821607 | |
1352378370 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352378370 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5Qm6gC | simonw 9599 | 2022-12-15T00:02:08Z | 2022-12-15T00:04:54Z | OWNER | I fixed this issue to help research this further: - https://github.com/simonw/datasette-ripgrep/issues/26 Now this search works: https://ripgrep.datasette.io/-/ripgrep?pattern=return+_error&literal=on&glob=datasette%2F** I wish I had this feature! - https://github.com/simonw/datasette-ripgrep/issues/24 Looks like I have both |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352371019 | https://github.com/simonw/datasette/issues/1957#issuecomment-1352371019 | https://api.github.com/repos/simonw/datasette/issues/1957 | IC_kwDOBm6k_c5Qm4tL | simonw 9599 | 2022-12-14T23:50:50Z | 2022-12-14T23:50:50Z | OWNER | One option: if any rows were truncated, show a button you can click to run the query again with truncation disabled - maybe with But... still want to truncate if the user runs a query that would return multiple MBs of HTML (assuming the other Datasette query limits don't stop that from happening). |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Reconsider row value truncation on query page 1497577017 | |
1352357322 | https://github.com/simonw/datasette/issues/1295#issuecomment-1352357322 | https://api.github.com/repos/simonw/datasette/issues/1295 | IC_kwDOBm6k_c5Qm1XK | simonw 9599 | 2022-12-14T23:28:49Z | 2022-12-14T23:28:49Z | OWNER | Related: - #1875 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Errors should have links to further information 855296937 | |
1352356356 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352356356 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5Qm1IE | simonw 9599 | 2022-12-14T23:27:25Z | 2022-12-14T23:28:16Z | OWNER | Also weird: errors returned by that mechanism look like this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352354927 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352354927 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5Qm0xv | simonw 9599 | 2022-12-14T23:25:06Z | 2022-12-14T23:25:14Z | OWNER | Looks like the code I've written for permission checking on Which uses this: Having two different patterns to return errors is bad, I should fix that. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352340518 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352340518 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5QmxQm | simonw 9599 | 2022-12-14T23:07:01Z | 2022-12-14T23:07:01Z | OWNER | Easiest fix would be to look for Not bullet-proof, so people might occasionally make JSON requests and get back an HTML error - but the documentation can tell people that they need to send those headers if they want to reliably get back JSON error messages. I'm happy with this as a solution. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352338620 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352338620 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5Qmwy8 | simonw 9599 | 2022-12-14T23:05:17Z | 2022-12-14T23:05:17Z | OWNER | Sniffing for a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352335503 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352335503 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5QmwCP | simonw 9599 | 2022-12-14T23:03:28Z | 2022-12-14T23:03:28Z | OWNER | This raises a more complicated issue At some point I'm likely to want to add an HTML interface for creating tables and inserting and updating rows. The obvious URLs for that are the same as for the JSON API: Those endpoints are currently POST only - and can return JSON all the time. If they start accepting form POSTs too they'll need to be able to accept form-encoded data and return HTML instead. That's OK - they can detect incoming JSON thanks to the I think it can still work though: I'll only set |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352331314 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352331314 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5QmvAy | simonw 9599 | 2022-12-14T22:59:36Z | 2022-12-14T22:59:36Z | OWNER | I'm going to prototype that up to see what it looks like. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352330825 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352330825 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5Qmu5J | simonw 9599 | 2022-12-14T22:58:51Z | 2022-12-14T22:59:27Z | OWNER | I need a way for those JSON endpoints to communicate back to the Since it gets the It's a bit of a cludge though! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352329027 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352329027 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5QmudD | simonw 9599 | 2022-12-14T22:56:24Z | 2022-12-14T22:57:19Z | OWNER | Most But that code triggers when the URL ends with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352075845 | https://github.com/simonw/datasette/issues/1956#issuecomment-1352075845 | https://api.github.com/repos/simonw/datasette/issues/1956 | IC_kwDOBm6k_c5QlwpF | simonw 9599 | 2022-12-14T19:57:17Z | 2022-12-14T19:58:22Z | OWNER | I'm going to test this using calls to I can add extra tests to https://github.com/simonw/datasette/blob/1a3dcf494376e32f7cff110c86a88e5b0a3f3924/tests/test_permissions.py#L605-L616 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Handle abbreviations properly in permission_allowed_actor_restrictions 1497288666 | |
1352070655 | https://github.com/simonw/datasette/issues/1956#issuecomment-1352070655 | https://api.github.com/repos/simonw/datasette/issues/1956 | IC_kwDOBm6k_c5QlvX_ | simonw 9599 | 2022-12-14T19:54:36Z | 2022-12-14T19:54:36Z | OWNER | Also this code should work with non-abbreviations too. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Handle abbreviations properly in permission_allowed_actor_restrictions 1497288666 | |
1352644276 | https://github.com/simonw/datasette/issues/1958#issuecomment-1352644276 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5Qn7a0 | davidhaley 11729897 | 2022-12-14T14:53:53Z | 2022-12-14T14:53:53Z | NONE | I don't have much experience with Python; however, I wonder if this print statement needs |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1352644274 | https://github.com/simonw/datasette/issues/1958#issuecomment-1352644274 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5Qn7ay | davidhaley 11729897 | 2022-12-14T14:19:24Z | 2022-12-14T14:19:24Z | NONE | Hmm, it appears after I kill the process with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1350415644 | https://github.com/simonw/datasette/issues/1952#issuecomment-1350415644 | https://api.github.com/repos/simonw/datasette/issues/1952 | IC_kwDOBm6k_c5QfbUc | simonw 9599 | 2022-12-14T05:22:59Z | 2022-12-14T05:22:59Z | OWNER | Non-memory named databases shouldn't show write actions, since those won't persist. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Improvements to /-/create-token restrictions interface 1495716243 | |
1350414961 | https://github.com/simonw/datasette/issues/1947#issuecomment-1350414961 | https://api.github.com/repos/simonw/datasette/issues/1947 | IC_kwDOBm6k_c5QfbJx | simonw 9599 | 2022-12-14T05:22:00Z | 2022-12-14T05:22:00Z | OWNER | I think the next big step for this feature is for me to actually use it to build a few things. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UI to create reduced scope tokens from the `/-/create-token` page 1493390939 | |
1350414402 | https://github.com/simonw/datasette/issues/1947#issuecomment-1350414402 | https://api.github.com/repos/simonw/datasette/issues/1947 | IC_kwDOBm6k_c5QfbBC | simonw 9599 | 2022-12-14T05:21:07Z | 2022-12-14T05:21:07Z | OWNER | It would be neat not to show write permissions against immutable databases too - and not hard from a performance perspective since it doesn't involve hundreds more permission checks. That will need permissions to grow a flag for if they need a mutable database though, which is a bigger job. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UI to create reduced scope tokens from the `/-/create-token` page 1493390939 | |
1350413555 | https://github.com/simonw/datasette/issues/1947#issuecomment-1350413555 | https://api.github.com/repos/simonw/datasette/issues/1947 | IC_kwDOBm6k_c5Qfazz | simonw 9599 | 2022-12-14T05:19:52Z | 2022-12-14T05:19:52Z | OWNER | Maybe I should have kept Yeah I think I should. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UI to create reduced scope tokens from the `/-/create-token` page 1493390939 | |
1350409537 | https://github.com/simonw/datasette/issues/1947#issuecomment-1350409537 | https://api.github.com/repos/simonw/datasette/issues/1947 | IC_kwDOBm6k_c5QfZ1B | simonw 9599 | 2022-12-14T05:14:16Z | 2022-12-14T05:14:16Z | OWNER | New interface now live at https://latest.datasette.io/-/create-token It shouldn't be showing |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UI to create reduced scope tokens from the `/-/create-token` page 1493390939 | |
1350402667 | https://github.com/simonw/datasette/issues/1947#issuecomment-1350402667 | https://api.github.com/repos/simonw/datasette/issues/1947 | IC_kwDOBm6k_c5QfYJr | simonw 9599 | 2022-12-14T05:05:10Z | 2022-12-14T05:05:10Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UI to create reduced scope tokens from the `/-/create-token` page 1493390939 | ||
1350401651 | https://github.com/simonw/datasette/issues/1947#issuecomment-1350401651 | https://api.github.com/repos/simonw/datasette/issues/1947 | IC_kwDOBm6k_c5QfX5z | simonw 9599 | 2022-12-14T05:03:59Z | 2022-12-14T05:03:59Z | OWNER | I shipped a working interface. Could still do with some extra tests. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UI to create reduced scope tokens from the `/-/create-token` page 1493390939 | |
1350293098 | https://github.com/simonw/datasette/issues/1951#issuecomment-1350293098 | https://api.github.com/repos/simonw/datasette/issues/1951 | IC_kwDOBm6k_c5Qe9Zq | simonw 9599 | 2022-12-14T02:43:44Z | 2022-12-14T02:43:44Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`datasette.create_token(...)` method for creating signed API tokens 1495431932 | ||
1350231654 | https://github.com/simonw/datasette/issues/1951#issuecomment-1350231654 | https://api.github.com/repos/simonw/datasette/issues/1951 | IC_kwDOBm6k_c5QeuZm | simonw 9599 | 2022-12-14T01:48:50Z | 2022-12-14T01:48:57Z | OWNER | I like that the word |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`datasette.create_token(...)` method for creating signed API tokens 1495431932 | |
1350222701 | https://github.com/simonw/datasette/issues/1951#issuecomment-1350222701 | https://api.github.com/repos/simonw/datasette/issues/1951 | IC_kwDOBm6k_c5QesNt | simonw 9599 | 2022-12-14T01:35:05Z | 2022-12-14T01:35:22Z | OWNER | Maybe this: ```python datasette.create_token("root", expires_after=3600, restrict_all=("view-query", "view-table")) token = datasette.create_token("root", expires_after=3600, restrict_database={ "fixtures": ("view-query",) }) token = datasette.create_token("root", expires_after=3600, restrict_resource={ "fixtures": { "facetable": ("insert-row", "update-row") } }) ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`datasette.create_token(...)` method for creating signed API tokens 1495431932 | |
1350220579 | https://github.com/simonw/datasette/issues/1951#issuecomment-1350220579 | https://api.github.com/repos/simonw/datasette/issues/1951 | IC_kwDOBm6k_c5Qersj | simonw 9599 | 2022-12-14T01:31:38Z | 2022-12-14T01:31:38Z | OWNER | The problem with Likewise, And the nested |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`datasette.create_token(...)` method for creating signed API tokens 1495431932 | |
1350218177 | https://github.com/simonw/datasette/issues/1951#issuecomment-1350218177 | https://api.github.com/repos/simonw/datasette/issues/1951 | IC_kwDOBm6k_c5QerHB | simonw 9599 | 2022-12-14T01:27:45Z | 2022-12-14T01:30:41Z | OWNER | Some sketches: ```python Token for root usertoken = datasette.create_token("root") Expiring in an hourtoken = datasette.create_token("root", expires_after=3600)
Limited to view-query and view-tabletoken = datasette.create_token("root", expires_after=3600, all=("view-query", "view-table")) I'm not sure about that all= nameLimits within a specific database:token = datasette.create_token("root", expires_after=3600, databases={ "fixtures": ("view-query",) }) And specific tables:token = datasette.create_token("root", expires_after=3600, tables={ "fixtures": { "facetable": ("insert-row", "update-row") } }) ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`datasette.create_token(...)` method for creating signed API tokens 1495431932 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user >30