github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/issues/878#issuecomment-1073037939 | https://api.github.com/repos/simonw/datasette/issues/878 | 1073037939 | IC_kwDOBm6k_c4_9UJz | 9599 | 2022-03-19T16:19:30Z | 2022-03-19T16:19:30Z | OWNER | On revisiting https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 a few months later I'm having second thoughts about using `@inject` on the `main()` method. But I still like the pattern as a way to resolve more complex cases like "to generate GeoJSON of the expanded view with labels, the label expansion code needs to run once at some before the GeoJSON formatting code does". So I'm going to stick with it a tiny bit longer, but maybe try to make it a lot more explicit when it's going to happen rather than having the main view methods themselves also use async DI. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-1001699559 | https://api.github.com/repos/simonw/datasette/issues/878 | 1001699559 | IC_kwDOBm6k_c47tLjn | 9599 | 2021-12-27T18:53:04Z | 2021-12-27T18:53:04Z | OWNER | I'm going to see if I can come up with the simplest possible version of this pattern for the `/-/metadata` and `/-/metadata.json` page, then try it for the database query page, before tackling the much more complex table page. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-973678931 | https://api.github.com/repos/simonw/datasette/issues/878 | 973678931 | IC_kwDOBm6k_c46CSlT | 9599 | 2021-11-19T02:51:17Z | 2021-11-19T02:51:17Z | OWNER | OK, I managed to get a table to render! Here's the code I used - I had to copy a LOT of stuff. https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 I'm going to move this work into a new, separate issue. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-973635157 | https://api.github.com/repos/simonw/datasette/issues/878 | 973635157 | IC_kwDOBm6k_c46CH5V | 9599 | 2021-11-19T01:07:08Z | 2021-11-19T01:07:08Z | OWNER | This exercise is proving so useful in getting my head around how the enormous and complex `TableView` class works again. Here's where I've got to now - I'm systematically working through the variables that are returned for HTML and for JSON copying across code to get it to work: ```python from datasette.database import QueryInterrupted from datasette.utils import escape_sqlite from datasette.utils.asgi import Response, NotFound, Forbidden from datasette.views.base import DatasetteError from datasette import hookimpl from asyncinject import AsyncInject, inject from pprint import pformat class Table(AsyncInject): @inject async def database(self, request, datasette): # TODO: all that nasty hash resolving stuff can go here db_name = request.url_vars["db_name"] try: db = datasette.databases[db_name] except KeyError: raise NotFound(f"Database '{db_name}' does not exist") return db @inject async def table_and_format(self, request, database, datasette): table_and_format = request.url_vars["table_and_format"] # TODO: be a lot smarter here if "." in table_and_format: return table_and_format.split(".", 2) else: return table_and_format, "html" @inject async def main(self, request, database, table_and_format, datasette): # TODO: if this is actually a canned query, dispatch to it table, format = table_and_format is_view = bool(await database.get_view_definition(table)) table_exists = bool(await database.table_exists(table)) if not is_view and not table_exists: raise NotFound(f"Table not found: {table}") await check_permissions( datasette, request, [ ("view-table", (database.name, table)), ("view-database", database.name), "view-instance", ], ) private =… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-973568285 | https://api.github.com/repos/simonw/datasette/issues/878 | 973568285 | IC_kwDOBm6k_c46B3kd | 9599 | 2021-11-19T00:29:20Z | 2021-11-19T00:29:20Z | OWNER | This is working! ```python from datasette.utils.asgi import Response from datasette import hookimpl import html from asyncinject import AsyncInject, inject class Table(AsyncInject): @inject async def database(self, request): return request.url_vars["db_name"] @inject async def main(self, request, database): return Response.html("Database: {}".format( html.escape(database) )) async def view(self, request): return await self.main(request=request) @hookimpl def register_routes(): return [ (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view), ] ``` This project will definitely show me if I actually like the `asyncinject` patterns or not. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-973564260 | https://api.github.com/repos/simonw/datasette/issues/878 | 973564260 | IC_kwDOBm6k_c46B2lk | 9599 | 2021-11-19T00:27:06Z | 2021-11-19T00:27:06Z | OWNER | Problem: the fancy `asyncinject` stuff inteferes with the fancy Datasette thing that introspects view functions to look for what parameters they take: ```python class Table(asyncinject.AsyncInjectAll): async def view(self, request): return Response.html("Hello from {}".format( html.escape(repr(request.url_vars)) )) @hookimpl def register_routes(): return [ (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view), ] ``` This failed with error: "Table.view() takes 1 positional argument but 2 were given" So I'm going to use `AsyncInject` and have the `view` function NOT use the `@inject` decorator. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-973554024 | https://api.github.com/repos/simonw/datasette/issues/878 | 973554024 | IC_kwDOBm6k_c46B0Fo | 9599 | 2021-11-19T00:21:20Z | 2021-11-19T00:21:20Z | OWNER | That's annoying: it looks like plugins can't use `register_routes()` to over-ride default routes within Datasette itself. This didn't work: ```python from datasette.utils.asgi import Response from datasette import hookimpl import html async def table(request): return Response.html("Hello from {}".format( html.escape(repr(request.url_vars)) )) @hookimpl def register_routes(): return [ (r"/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", table), ] ``` I'll use a `/t/` prefix for the moment, but this is probably something I'll fix in Datasette itself later. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-973542284 | https://api.github.com/repos/simonw/datasette/issues/878 | 973542284 | IC_kwDOBm6k_c46BxOM | 9599 | 2021-11-19T00:16:44Z | 2021-11-19T00:16:44Z | OWNER | ``` Development % cookiecutter gh:simonw/datasette-plugin You've downloaded /Users/simon/.cookiecutters/datasette-plugin before. Is it okay to delete and re-download it? [yes]: yes plugin_name []: table-new description []: New implementation of TableView, see https://github.com/simonw/datasette/issues/878 hyphenated [table-new]: underscored [table_new]: github_username []: simonw author_name []: Simon Willison include_static_directory []: include_templates_directory []: ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-973527870 | https://api.github.com/repos/simonw/datasette/issues/878 | 973527870 | IC_kwDOBm6k_c46Bts- | 9599 | 2021-11-19T00:13:43Z | 2021-11-19T00:13:43Z | OWNER | New plan: I'm going to build a brand new implementation of `TableView` starting out as a plugin, using the `register_routes()` plugin hook. It will reuse the existing HTML template but will be a completely new Python implementation, based on `asyncinject`. I'm going to start by just getting the table to show up on the page - then I'll add faceting, suggested facets, filters and so-on. Bonus: I'm going to see if I can get it to work for arbitrary SQL queries too (stretch goal). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-971209475 | https://api.github.com/repos/simonw/datasette/issues/878 | 971209475 | IC_kwDOBm6k_c4543sD | 9599 | 2021-11-17T05:41:42Z | 2021-11-17T05:41:42Z | OWNER | I'm going to build a brand new implementation of the `TableView` class that doesn't subclass `BaseView` at all, instead using `asyncinject`. If I'm lucky that will clean up the grungiest part of the codebase. I can maybe even run the tests against old `TableView` and `TableView2` to check that they behave the same. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-971057553 | https://api.github.com/repos/simonw/datasette/issues/878 | 971057553 | IC_kwDOBm6k_c454SmR | 9599 | 2021-11-17T01:40:45Z | 2021-11-17T01:40:45Z | OWNER | I shipped that code as a new library, `asyncinject`: https://pypi.org/project/asyncinject/ - I'll open a new PR to attempt to refactor `TableView` to use it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-970712713 | https://api.github.com/repos/simonw/datasette/issues/878 | 970712713 | IC_kwDOBm6k_c452-aJ | 9599 | 2021-11-16T21:54:33Z | 2021-11-16T21:54:33Z | OWNER | I'm going to continue working on this in a PR. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-970705738 | https://api.github.com/repos/simonw/datasette/issues/878 | 970705738 | IC_kwDOBm6k_c4528tK | 9599 | 2021-11-16T21:44:31Z | 2021-11-16T21:44:31Z | OWNER | Wrote a TIL about what I learned using `TopologicalSorter`: https://til.simonwillison.net/python/graphlib-topologicalsorter | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-970673085 | https://api.github.com/repos/simonw/datasette/issues/878 | 970673085 | IC_kwDOBm6k_c4520u9 | 9599 | 2021-11-16T20:58:24Z | 2021-11-16T20:58:24Z | OWNER | New test: ```python class Complex(AsyncBase): def __init__(self): self.log = [] async def d(self): await asyncio.sleep(random() * 0.1) print("LOG: d") self.log.append("d") async def c(self): await asyncio.sleep(random() * 0.1) print("LOG: c") self.log.append("c") async def b(self, c, d): print("LOG: b") self.log.append("b") async def a(self, b, c): print("LOG: a") self.log.append("a") async def go(self, a): print("LOG: go") self.log.append("go") return self.log @pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once assert tuple(result) in ( # c and d could happen in either order ("c", "d", "b", "a", "go"), ("d", "c", "b", "a", "go"), ) ``` And this code passes it: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != "self" and not p.startswith("_") } for key, method in _registry.items() } new_attrs["_graph"] = graph return super().__new__(cls, name, bases, new_attrs) def make_met… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-970660299 | https://api.github.com/repos/simonw/datasette/issues/878 | 970660299 | IC_kwDOBm6k_c452xnL | 9599 | 2021-11-16T20:39:43Z | 2021-11-16T20:42:27Z | OWNER | But that does seem to be the plan that `TopographicalSorter` provides: ```python graph = {"go": {"a"}, "a": {"b", "c"}, "b": {"c", "d"}} ts = TopologicalSorter(graph) ts.prepare() while ts.is_active(): nodes = ts.get_ready() print(nodes) ts.done(*nodes) ``` Outputs: ``` ('c', 'd') ('b',) ('a',) ('go',) ``` Also: ```python graph = {"go": {"d", "e", "f"}, "d": {"b", "c"}, "b": {"c"}} ts = TopologicalSorter(graph) ts.prepare() while ts.is_active(): nodes = ts.get_ready() print(nodes) ts.done(*nodes) ``` Gives: ``` ('e', 'f', 'c') ('b',) ('d',) ('go',) ``` I'm confident that `TopologicalSorter` is the way to do this. I think I need to rewrite my code to call it once to get that plan, then `await asyncio.gather(*nodes)` in turn to execute it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-970657874 | https://api.github.com/repos/simonw/datasette/issues/878 | 970657874 | IC_kwDOBm6k_c452xBS | 9599 | 2021-11-16T20:36:01Z | 2021-11-16T20:36:01Z | OWNER | My goal here is to calculate the most efficient way to resolve the different nodes, running them in parallel where possible. So for this class: ```python class Complex(AsyncBase): async def d(self): pass async def c(self): pass async def b(self, c, d): pass async def a(self, b, c): pass async def go(self, a): pass ``` A call to `go()` should do this: - `c` and `d` in parallel - `b` - `a` - `go` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-970655927 | https://api.github.com/repos/simonw/datasette/issues/878 | 970655927 | IC_kwDOBm6k_c452wi3 | 9599 | 2021-11-16T20:33:11Z | 2021-11-16T20:33:11Z | OWNER | What should be happening here instead is it should resolve the full graph and notice that `c` is depended on by both `b` and `a` - so it should run `c` first, then run the next ones in parallel. So maybe the algorithm I'm inheriting from https://docs.python.org/3/library/graphlib.html isn't the correct algorithm? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-970655304 | https://api.github.com/repos/simonw/datasette/issues/878 | 970655304 | IC_kwDOBm6k_c452wZI | 9599 | 2021-11-16T20:32:16Z | 2021-11-16T20:32:16Z | OWNER | This code is really fiddly. I just got to this version: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != "self" and not p.startswith("_") } for key, method in _registry.items() } new_attrs["_graph"] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, _results=None, **kwargs): print("inner - _results=", _results) parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != "self"] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), "The following DI parameters could not be found in the registry: {}".format( missing ) results = {} results.update(kwargs) if to_resolve: resolved_parameters = await self.resolve(to_resolve, _results) results.update(resolved_parameters) return_value = await method(self, **results) if _results is not None: _res… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-970624197 | https://api.github.com/repos/simonw/datasette/issues/878 | 970624197 | IC_kwDOBm6k_c452ozF | 9599 | 2021-11-16T19:49:05Z | 2021-11-16T19:49:05Z | OWNER | Here's the latest version of my weird dependency injection async class: ```python import inspect class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Topological sort of _registry by parameter dependencies graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != "self" and not p.startswith("_") } for key, method in _registry.items() } new_attrs["_graph"] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, **kwargs): parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != "self"] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), "The following DI parameters could not be found in the registry: {}".format( missing ) results = {} results.update(kwargs) results.update(await self.resolve(to_resolve)) return await method(self, **results) return inner bad = [0] class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names): print(" resolve({})".format(names)) results = {} # Resolve them in the correct order ts = TopologicalSorter() ts2 = TopologicalSorter() print(" names = ", names) print(" s… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-803473015 | https://api.github.com/repos/simonw/datasette/issues/878 | 803473015 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MzAxNQ== | 9599 | 2021-03-20T22:33:05Z | 2021-03-20T22:33:05Z | OWNER | Things this mechanism needs to be able to support: - Returning a default JSON representation - Defining "extra" JSON representations blocks, which can be requested using `?_extra=` - Returning rendered HTML, based on the default JSON + one or more extras + a template - Using Datasette output renderers to return e.g. CSV data - Potentially also supporting streaming output renderers for streaming CSV/TSV/JSON-nl etc | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-803472595 | https://api.github.com/repos/simonw/datasette/issues/878 | 803472595 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MjU5NQ== | 9599 | 2021-03-20T22:28:12Z | 2021-03-20T22:28:12Z | OWNER | Another idea I had: a view is a class that takes the `datasette` instance in its constructor, and defines a `__call__` method that accepts a request and returns a response. Except `await __call__` looks like it might be a bit messy, discussion in https://github.com/encode/starlette/issues/886 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-803471917 | https://api.github.com/repos/simonw/datasette/issues/878 | 803471917 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MTkxNw== | 9599 | 2021-03-20T22:21:33Z | 2021-03-20T22:21:33Z | OWNER | This has been blocking things for too long. If this becomes a documented pattern, things like adding a JSON output to https://github.com/dogsheep/dogsheep-beta becomes easier too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-709503359 | https://api.github.com/repos/simonw/datasette/issues/878 | 709503359 | MDEyOklzc3VlQ29tbWVudDcwOTUwMzM1OQ== | 9599 | 2020-10-15T18:15:28Z | 2020-10-15T18:15:28Z | OWNER | I think this is blocking #619 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 | |
https://github.com/simonw/datasette/issues/878#issuecomment-709502889 | https://api.github.com/repos/simonw/datasette/issues/878 | 709502889 | MDEyOklzc3VlQ29tbWVudDcwOTUwMjg4OQ== | 9599 | 2020-10-15T18:14:34Z | 2020-10-15T18:14:34Z | OWNER | The `BaseView` class does this for Datasette internals at the moment, but I'm not convinced it works as well as it could. I'd like to turn this into a class that is documented and available to plugins as well. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648435885 |