18 rows where issue = 717746043 sorted by updated_at descending

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

user

author_association

issue

  • datasette.client internal requests mechanism · 18
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
706269271 https://github.com/simonw/datasette/pull/1000#issuecomment-706269271 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNjI2OTI3MQ== simonw 9599 2020-10-09T16:08:49Z 2020-10-09T16:08:49Z OWNER

I'm going to document this in a separate issue.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705890365 https://github.com/simonw/datasette/pull/1000#issuecomment-705890365 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTg5MDM2NQ== codecov[bot] 22429695 2020-10-09T00:03:29Z 2020-10-09T16:07:03Z NONE

Codecov Report

Merging #1000 into main will increase coverage by 0.15%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1000      +/-   ##
==========================================
+ Coverage   84.37%   84.52%   +0.15%     
==========================================
  Files          28       28              
  Lines        3871     3878       +7     
==========================================
+ Hits         3266     3278      +12     
+ Misses        605      600       -5     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/app.py</td> <td>96.34% <100.00%> (+0.02%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/cli.py</td> <td>74.35% <100.00%> (ø)</td> <td></td> </tr> <tr> <td>datasette/utils/testing.py</td> <td>95.16% <100.00%> (-4.84%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/views/base.py</td> <td>93.94% <100.00%> (+0.11%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/utils/asgi.py</td> <td>91.92% <0.00%> (ø)</td> <td></td> </tr> <tr> <td>datasette/views/special.py</td> <td>93.51% <0.00%> (+8.33%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7249ac5...8a80c79. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
706263157 https://github.com/simonw/datasette/pull/1000#issuecomment-706263157 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNjI2MzE1Nw== simonw 9599 2020-10-09T15:57:15Z 2020-10-09T15:57:15Z OWNER

My httpx pull request adding raw_path support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.

I'm going to mark these tests as xfail so I can land this change - I'll remove that once an httpx release comes out that I can use to get the tests passing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705946360 https://github.com/simonw/datasette/pull/1000#issuecomment-705946360 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTk0NjM2MA== simonw 9599 2020-10-09T03:28:08Z 2020-10-09T03:28:08Z OWNER

Here's where httpx sets up the ASGI scope: https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L82-L97

        # ASGI scope.
        scheme, host, port, full_path = url
        path, _, query = full_path.partition(b"?")
        scope = {
            "type": "http",
            "asgi": {"version": "3.0"},
            "http_version": "1.1",
            "method": method.decode(),
            "headers": [(k.lower(), v) for (k, v) in headers],
            "scheme": scheme.decode("ascii"),
            "path": unquote(path.decode("ascii")),
            "query_string": query,
            "server": (host.decode("ascii"), port),
            "client": self.client,
            "root_path": self.root_path,
        }

Sure enough, it doesn't set the raw_path.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705946120 https://github.com/simonw/datasette/pull/1000#issuecomment-705946120 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTk0NjEyMA== simonw 9599 2020-10-09T03:27:05Z 2020-10-09T03:27:05Z OWNER

I may need to fuss around with how the httpx client sends things to the ASGI app.

https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L26 is relevant:

Alternatively, you can setup the transport instance explicitly.
This allows you to include any additional configuration arguments specific
to the ASGITransport class:
```
transport = httpx.ASGITransport(
    app=app,
    root_path="/submount",
    client=("1.2.3.4", 123)
)
client = httpx.AsyncClient(transport=transport)
```
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705945591 https://github.com/simonw/datasette/pull/1000#issuecomment-705945591 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTk0NTU5MQ== simonw 9599 2020-10-09T03:24:48Z 2020-10-09T03:24:48Z OWNER

I'm testing this with a print(scope) and pytest -k test_table_with_slashes_in_name -s.

Against the main branch:

{'type': 'http', 'http_version': '1.0', 'method': 'GET', 'path': '/fixtures/table/with/slashes.csv', 'raw_path': b'/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'headers': [[b'host', b'localhost']], 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x10e2e6040>}

Against the broken branch:

tests/test_api.py {'type': 'http', 'asgi': {'version': '3.0'}, 'http_version': '1.1', 'method': 'GET', 'headers': [(b'host', b'localhost'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'user-agent', b'python-httpx/0.15.0')], 'scheme': 'http', 'path': '/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'server': ('localhost', None), 'client': ('127.0.0.1', 123), 'root_path': '', 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x109e0eca0>}

This is on my laptop though so both of those pass the tests.

Key difference: the httpx version doesn't set a raw_path at all. BUT.. it does set path and sets it to '/fixtures/table%2Fwith%2Fslashes.csv'

The non-httpx version sets raw_path to b'/fixtures/table%2Fwith%2Fslashes.csv' and path to '/fixtures/table/with/slashes.csv'.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705941580 https://github.com/simonw/datasette/pull/1000#issuecomment-705941580 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTk0MTU4MA== simonw 9599 2020-10-09T03:08:43Z 2020-10-09T03:08:43Z OWNER

Most likely reason for those failures is that path and raw_path are not being simulated correctly. I used to do those here:

https://github.com/simonw/datasette/blob/402cf870b7d65f9b5fba9e23aa99433294bd4523/datasette/utils/testing.py#L116-L125

But now I'm delegating that to httpx to handle.

WEIRD that it passes on my laptop but fails in GitHub Actions CI though.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705940507 https://github.com/simonw/datasette/pull/1000#issuecomment-705940507 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTk0MDUwNw== simonw 9599 2020-10-09T03:04:15Z 2020-10-09T03:04:15Z OWNER

This is really weird: new set of test failures that I wasn't seeing before, and those tests aren't failing on my laptop:

=========================== short test summary info ============================
FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200
FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200
FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert...
FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5]
FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html]
================== 5 failed, 738 passed in 194.73s (0:03:14) ===================
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705937696 https://github.com/simonw/datasette/pull/1000#issuecomment-705937696 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkzNzY5Ng== simonw 9599 2020-10-09T02:52:53Z 2020-10-09T02:52:53Z OWNER

These failures are giving me a severe "how did this ever work in the first place?" vibe:

FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/]

I have a fix for them, no idea why they weren't already failing though.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705926445 https://github.com/simonw/datasette/pull/1000#issuecomment-705926445 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkyNjQ0NQ== simonw 9599 2020-10-09T02:15:38Z 2020-10-09T02:15:38Z OWNER

FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages'

That one is caused by response.cookies skipping cookies that were set to the empty string. Same fix as this: https://github.com/simonw/datasette/blob/a1687351fb75b01f737fda4ad07e0781029de05c/tests/test_auth.py#L90-L95

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705926035 https://github.com/simonw/datasette/pull/1000#issuecomment-705926035 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkyNjAzNQ== simonw 9599 2020-10-09T02:14:14Z 2020-10-09T02:14:14Z OWNER

Still need to handle these six failing tests:

FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'elemen...
FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', '...
FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<...
FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<p cla...
FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages'
FAILED tests/test_plugins.py::test_hook_register_magic_parameters - AssertionError: assert [{'line': '1.0', 'rowid': 1}] == [{'line': '1.1', 'rowid': 1}]
=========================================================== 6 failed, 731 passed, 6 warnings in 129.17s (0:02:09) ===========================================================
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705921006 https://github.com/simonw/datasette/pull/1000#issuecomment-705921006 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkyMTAwNg== simonw 9599 2020-10-09T01:55:01Z 2020-10-09T01:55:01Z OWNER

With the single client that is reused for all tests:

% time pytest tests/test_api.py
...
6.73s user 9.91s system 81% cpu 20.365 total

After switching back to this class:

class DatasetteClient:
    def __init__(self, ds):
        self.app = ds.app()

    def _fix(self, path):
        if path.startswith("/"):
            path = "http://localhost{}".format(path)
        return path

    async def get(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.get(self._fix(path), **kwargs)

    async def options(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.options(self._fix(path), **kwargs)

    async def head(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.head(self._fix(path), **kwargs)

    async def post(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.post(self._fix(path), **kwargs)

    async def put(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.put(self._fix(path), **kwargs)

    async def patch(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.patch(self._fix(path), **kwargs)

    async def delete(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.delete(self._fix(path), **kwargs)

    async def request(self, method, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.request(method, self._fix(path), **kwargs)

The time taken is:

% time pytest tests/test_api.py
...
7.26s user 10.02s system 82% cpu 21.014 total

That's close enough that I don't feel I need to investigate this further.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705920228 https://github.com/simonw/datasette/pull/1000#issuecomment-705920228 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkyMDIyOA== simonw 9599 2020-10-09T01:51:44Z 2020-10-09T01:51:44Z OWNER

I'm going to switch back to having each request run through a new client. I'm worried about the impact on test performance though. I'll run a microbenchmark before and after.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705920055 https://github.com/simonw/datasette/pull/1000#issuecomment-705920055 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkyMDA1NQ== simonw 9599 2020-10-09T01:51:05Z 2020-10-09T01:51:05Z OWNER

The topic of disabling cookie persistence is discussed a little here: https://github.com/encode/httpx/issues/422#issuecomment-537906693

We have a cookiejar abstraction, I think setting it to an always-empty jar like you describe is best. :)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705918844 https://github.com/simonw/datasette/pull/1000#issuecomment-705918844 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkxODg0NA== simonw 9599 2020-10-09T01:46:06Z 2020-10-09T01:46:06Z OWNER

For this failing test I'm suspicious that the AsyncClient may be persisting cookies in between requests:

    def test_actor_cookie(app_client):
        "A valid actor cookie sets request.scope['actor']"
        cookie = app_client.actor_cookie({"id": "test"})
        response = app_client.get("/", cookies={"ds_actor": cookie})
>       assert {"id": "test"} == app_client.ds._last_request.scope["actor"]
E       AssertionError: assert {'id': 'test'} == {'id': 'root'}
E         Differing items:
E         {'id': 'test'} != {'id': 'root'}
E         Use -v to get the full diff
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705902902 https://github.com/simonw/datasette/pull/1000#issuecomment-705902902 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkwMjkwMg== simonw 9599 2020-10-09T00:50:49Z 2020-10-09T00:50:49Z OWNER

Almost all of the tests are passing:

=========================== short test summary info ============================
FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200
FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200
FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert...
FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5]
FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html]
FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/]
FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyE...
FAILED tests/test_plugins.py::test_hook_register_magic_parameters - Assertion...
============ 11 failed, 718 passed, 6 warnings in 225.77s (0:03:45) ============
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705899629 https://github.com/simonw/datasette/pull/1000#issuecomment-705899629 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTg5OTYyOQ== simonw 9599 2020-10-09T00:37:02Z 2020-10-09T00:37:02Z OWNER

I'm going to route the existing TestClient through this mechanism to exercise it during the tests.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  
705889120 https://github.com/simonw/datasette/pull/1000#issuecomment-705889120 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTg4OTEyMA== simonw 9599 2020-10-08T23:59:01Z 2020-10-08T23:59:01Z OWNER

Needs tests and documentation.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.client internal requests mechanism 717746043  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);