issue_comments
10,495 rows sorted by updated_at descending
This data as json, CSV (advanced)
issue >30
- Show column metadata plus links for foreign keys on arbitrary query results 50
- Redesign default .json format 48
- JavaScript plugin hooks mechanism similar to pluggy 47
- Updated Dockerfile with SpatiaLite version 5.0 45
- Port Datasette to ASGI 42
- Authentication (and permissions) as a core concept 40
- await datasette.client.get(path) mechanism for executing internal requests 33
- Maintain an in-memory SQLite table of connected databases and their tables 32
- Ability to sort (and paginate) by column 31
- link_or_copy_directory() error - Invalid cross-device link 28
- Export to CSV 27
- base_url configuration setting 27
- Documentation with recommendations on running Datasette in production without using Docker 27
- Support cross-database joins 26
- Ability for a canned query to write to the database 26
- table.transform() method for advanced alter table 26
- Proof of concept for Datasette on AWS Lambda with EFS 25
- WIP: Add Gmail takeout mbox import 25
- Redesign register_output_renderer callback 24
- New pattern for views that return either JSON or HTML, available for plugins 24
- "datasette insert" command and plugin hook 23
- Datasette Plugins 22
- .json and .csv exports fail to apply base_url 22
- Idea: import CSV to memory, run SQL, export in a single command 22
- table.extract(...) method and "sqlite-utils extract" command 21
- Database page loads too slowly with many large tables (due to table counts) 21
- ?sort=colname~numeric to sort by by column cast to real 21
- Handle spatialite geometry columns better 20
- "flash messages" mechanism 20
- Move CI to GitHub Issues 20
- …
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
974562942 | https://github.com/simonw/datasette/issues/1519#issuecomment-974562942 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FqZ- | simonw 9599 | 2021-11-20T00:59:32Z | 2021-11-20T00:59:32Z | OWNER | Ouch a nasty bug crept through there - https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/compound_three_primary_keys says
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974561593 | https://github.com/simonw/datasette/issues/1519#issuecomment-974561593 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FqE5 | simonw 9599 | 2021-11-20T00:53:19Z | 2021-11-20T00:53:19Z | OWNER | Adding that test found (I hope!) all of the remaining |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974559176 | https://github.com/simonw/datasette/issues/1519#issuecomment-974559176 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FpfI | simonw 9599 | 2021-11-20T00:42:08Z | 2021-11-20T00:42:08Z | OWNER |
This worked, I managed to get the tests to fail! Here's the change I made: ```diff diff --git a/tests/test_html.py b/tests/test_html.py index f24165b..dbdfe59 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -1614,12 +1614,19 @@ def test_metadata_sort_desc(app_client): "/fixtures/compound_three_primary_keys/a,a,a", "/fixtures/paginated_view", "/fixtures/facetable", + "/fixtures?sql=select+1", ], ) -def test_base_url_config(app_client_base_url_prefix, path): +@pytest.mark.parametrize("use_prefix", (True, False)) +def test_base_url_config(app_client_base_url_prefix, path, use_prefix): client = app_client_base_url_prefix - response = client.get("/prefix/" + path.lstrip("/")) + path_to_get = path + if use_prefix: + path_to_get = "/prefix/" + path.lstrip("/") + response = client.get(path_to_get) soup = Soup(response.body, "html.parser") + if path == "/fixtures?sql=select+1": + assert False for el in soup.findAll(["a", "link", "script"]): if "href" in el.attrs: href = el["href"] @@ -1642,11 +1649,12 @@ def test_base_url_config(app_client_base_url_prefix, path): # If this has been made absolute it may start http://localhost/ if href.startswith("http://localhost/"): href = href[len("http://localost/") :] - assert href.startswith("/prefix/"), { + assert href.startswith("/prefix/"), json.dumps({ "path": path, + "path_to_get": path_to_get, "href_or_src": href, "element_parent": str(el.parent), - } + }, indent=4, default=repr) def test_base_url_affects_metadata_extra_css_urls(app_client_base_url_prefix): ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974558267 | https://github.com/simonw/datasette/issues/1519#issuecomment-974558267 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FpQ7 | simonw 9599 | 2021-11-20T00:37:57Z | 2021-11-20T00:37:57Z | OWNER | Thanks to #1522 I have a live demo that exhibits this bug now: https://apache-proxy-demo.datasette.io/prefix/fixtures/attraction_characteristic |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974558076 | https://github.com/simonw/datasette/issues/1522#issuecomment-974558076 | https://api.github.com/repos/simonw/datasette/issues/1522 | IC_kwDOBm6k_c46FpN8 | simonw 9599 | 2021-11-20T00:36:56Z | 2021-11-20T00:36:56Z | OWNER | That 503 error is really frustrating: I have a deploy running at https://apache-proxy-demo.datasette.io/prefix/ and after a fresh deploy it serves 503 errors for quite a while - then eventually starts working. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Deploy a live instance of demos/apache-proxy 1058896236 | |
974557766 | https://github.com/simonw/datasette/issues/1522#issuecomment-974557766 | https://api.github.com/repos/simonw/datasette/issues/1522 | IC_kwDOBm6k_c46FpJG | simonw 9599 | 2021-11-20T00:35:25Z | 2021-11-20T00:35:25Z | OWNER | Wrote a TIL about |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Deploy a live instance of demos/apache-proxy 1058896236 | |
974542348 | https://github.com/simonw/datasette/issues/1522#issuecomment-974542348 | https://api.github.com/repos/simonw/datasette/issues/1522 | IC_kwDOBm6k_c46FlYM | simonw 9599 | 2021-11-19T23:41:47Z | 2021-11-19T23:44:07Z | OWNER | Do I have to use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Deploy a live instance of demos/apache-proxy 1058896236 | |
974541971 | https://github.com/simonw/datasette/issues/1522#issuecomment-974541971 | https://api.github.com/repos/simonw/datasette/issues/1522 | IC_kwDOBm6k_c46FlST | simonw 9599 | 2021-11-19T23:40:32Z | 2021-11-19T23:40:32Z | OWNER | I want to be able to use build arguments to specify which commit version or branch of Datasette to deploy. This is proving hard to work out. I have this in my Dockerfile now: ``` ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip ``` Which works locally:
But I can't figure out the right incantation to pass to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Deploy a live instance of demos/apache-proxy 1058896236 | |
974523569 | https://github.com/simonw/datasette/issues/1522#issuecomment-974523569 | https://api.github.com/repos/simonw/datasette/issues/1522 | IC_kwDOBm6k_c46Fgyx | simonw 9599 | 2021-11-19T22:51:10Z | 2021-11-19T22:51:10Z | OWNER | I wan a GitHub Action which I can manually activate to deploy a new version of that demo... and I want it to bake in the latest release of Datasette so I can use it to demonstrate bug fixes. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Deploy a live instance of demos/apache-proxy 1058896236 | |
974523297 | https://github.com/simonw/datasette/issues/1522#issuecomment-974523297 | https://api.github.com/repos/simonw/datasette/issues/1522 | IC_kwDOBm6k_c46Fguh | simonw 9599 | 2021-11-19T22:50:31Z | 2021-11-19T22:50:31Z | OWNER | Demo code is now at: https://github.com/simonw/datasette/tree/main/demos/apache-proxy |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Deploy a live instance of demos/apache-proxy 1058896236 | |
974521687 | https://github.com/simonw/datasette/issues/1522#issuecomment-974521687 | https://api.github.com/repos/simonw/datasette/issues/1522 | IC_kwDOBm6k_c46FgVX | simonw 9599 | 2021-11-19T22:46:26Z | 2021-11-19T22:46:26Z | OWNER | Oh weird, it started working: https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/sortable |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Deploy a live instance of demos/apache-proxy 1058896236 | |
974506401 | https://github.com/simonw/datasette/issues/1522#issuecomment-974506401 | https://api.github.com/repos/simonw/datasette/issues/1522 | IC_kwDOBm6k_c46Fcmh | simonw 9599 | 2021-11-19T22:11:51Z | 2021-11-19T22:11:51Z | OWNER | This is frustrating: I have the following Dockerfile: ```dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini Append this to the end of the default httpd.conf fileRUN echo $'ServerName localhost\n\ \n\ <Proxy *>\n\ Order deny,allow\n\ Allow from all\n\ </Proxy>\n\ \n\ ProxyPass /prefix/ http://localhost:8001/\n\ Header add X-Proxied-By "Apache2"' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url "/prefix/" -h 0.0.0.0 -p 8001 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80
ENTRYPOINT ["/tini", "--", "/app/start.sh"]
!/bin/bashhttps://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-runNAME="datasette-apache-proxy-demo" PROJECT=$(gcloud config get-value project) IMAGE="gcr.io/$PROJECT/$NAME" gcloud builds submit --tag $IMAGE
gcloud run deploy \
--allow-unauthenticated \
--platform=managed \
--image $IMAGE $NAME \
--port 80
Cloud Run logs: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Deploy a live instance of demos/apache-proxy 1058896236 | |
974478126 | https://github.com/simonw/datasette/issues/1519#issuecomment-974478126 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FVsu | simonw 9599 | 2021-11-19T21:16:36Z | 2021-11-19T21:16:36Z | OWNER | In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974477465 | https://github.com/simonw/datasette/issues/1519#issuecomment-974477465 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FViZ | simonw 9599 | 2021-11-19T21:15:30Z | 2021-11-19T21:15:30Z | OWNER | I think what's happening here is Apache is actually making a request to This is pretty confusing! I think Datasette should ONLY reply to But shipping that change could break existing deployments. Maybe that should be a breaking change for 1.0. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974450232 | https://github.com/simonw/datasette/issues/1519#issuecomment-974450232 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FO44 | simonw 9599 | 2021-11-19T20:41:53Z | 2021-11-19T20:42:19Z | OWNER | https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration says I should use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974447950 | https://github.com/simonw/datasette/issues/1519#issuecomment-974447950 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FOVO | simonw 9599 | 2021-11-19T20:40:19Z | 2021-11-19T20:40:19Z | OWNER | Figured it out! The test is not an accurate recreation of what is happening, because it doesn't simulate a request with a path of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974435661 | https://github.com/simonw/datasette/issues/1522#issuecomment-974435661 | https://api.github.com/repos/simonw/datasette/issues/1522 | IC_kwDOBm6k_c46FLVN | simonw 9599 | 2021-11-19T20:33:42Z | 2021-11-19T20:33:42Z | OWNER | Should just be a case of deploying this ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini Append this to the end of the default httpd.conf fileRUN echo $'ServerName localhost\n\ \n\ <Proxy *>\n\ Order deny,allow\n\ Allow from all\n\ </Proxy>\n\ \n\ ProxyPass /foo/bar/ http://localhost:9000/\n\ Header add X-Proxied-By "Apache2"' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url "/foo/bar/" -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT ["/tini", "--", "/app/start.sh"] ``` I can follow this TIL: https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Deploy a live instance of demos/apache-proxy 1058896236 | |
974433520 | https://github.com/simonw/datasette/issues/1521#issuecomment-974433520 | https://api.github.com/repos/simonw/datasette/issues/1521 | IC_kwDOBm6k_c46FKzw | simonw 9599 | 2021-11-19T20:32:29Z | 2021-11-19T20:32:29Z | OWNER | This configuration works great. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Docker configuration for exercising Datasette behind Apache mod_proxy 1058815557 | |
974433320 | https://github.com/simonw/datasette/issues/1519#issuecomment-974433320 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FKwo | simonw 9599 | 2021-11-19T20:32:04Z | 2021-11-19T20:32:04Z | OWNER | Still not clear why the tests pass but the live example fails. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974433206 | https://github.com/simonw/datasette/issues/1519#issuecomment-974433206 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FKu2 | simonw 9599 | 2021-11-19T20:31:52Z | 2021-11-19T20:31:52Z | OWNER | Modified my
And now the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974422829 | https://github.com/simonw/datasette/issues/1519#issuecomment-974422829 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FIMt | simonw 9599 | 2021-11-19T20:26:35Z | 2021-11-19T20:26:35Z | OWNER | In the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974420619 | https://github.com/simonw/datasette/issues/1519#issuecomment-974420619 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FHqL | simonw 9599 | 2021-11-19T20:25:19Z | 2021-11-19T20:25:19Z | OWNER | The implementations of https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/init.py#L228-L254 https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/init.py#L710-L729 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974418496 | https://github.com/simonw/datasette/issues/1519#issuecomment-974418496 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FHJA | simonw 9599 | 2021-11-19T20:24:16Z | 2021-11-19T20:24:16Z | OWNER | Here's the code that generates And here's the code for And for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974398399 | https://github.com/simonw/datasette/issues/1519#issuecomment-974398399 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FCO_ | simonw 9599 | 2021-11-19T20:08:20Z | 2021-11-19T20:22:02Z | OWNER | The relevant test is this one: https://github.com/simonw/datasette/blob/30255055150d7bc0affc8156adc18295495020ff/tests/test_html.py#L1608-L1649 I modified that test to add
This data as json, testall, testnone, testresponse, CSV
Something very weird is going on here. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974405016 | https://github.com/simonw/datasette/issues/1519#issuecomment-974405016 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FD2Y | simonw 9599 | 2021-11-19T20:14:19Z | 2021-11-19T20:15:05Z | OWNER | I added
And it's really strange that the bug doesn't show up in the tests. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974391204 | https://github.com/simonw/datasette/issues/1519#issuecomment-974391204 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FAek | simonw 9599 | 2021-11-19T20:02:41Z | 2021-11-19T20:02:41Z | OWNER | Bug confirmed: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974389472 | https://github.com/simonw/datasette/issues/1519#issuecomment-974389472 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46FADg | simonw 9599 | 2021-11-19T20:01:02Z | 2021-11-19T20:01:02Z | OWNER | I now have a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974388295 | https://github.com/simonw/datasette/issues/1521#issuecomment-974388295 | https://api.github.com/repos/simonw/datasette/issues/1521 | IC_kwDOBm6k_c46E_xH | simonw 9599 | 2021-11-19T20:00:06Z | 2021-11-19T20:00:06Z | OWNER | And this is the version that proxies to a ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini Append this to the end of the default httpd.conf fileRUN echo $'ServerName localhost\n\ \n\ <Proxy *>\n\ Order deny,allow\n\ Allow from all\n\ </Proxy>\n\ \n\ ProxyPass /foo/bar/ http://localhost:9000/\n\ Header add X-Proxied-By "Apache2"' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url "/foo/bar/" -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT ["/tini", "--", "/app/start.sh"] ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Docker configuration for exercising Datasette behind Apache mod_proxy 1058815557 | |
974380798 | https://github.com/simonw/datasette/issues/1521#issuecomment-974380798 | https://api.github.com/repos/simonw/datasette/issues/1521 | IC_kwDOBm6k_c46E97- | simonw 9599 | 2021-11-19T19:54:26Z | 2021-11-19T19:54:26Z | OWNER | Got it working! Here's a ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini Append this to the end of the default httpd.conf fileRUN echo $'ServerName localhost\n\ \n\ <Proxy *>\n\ Order deny,allow\n\ Allow from all\n\ </Proxy>\n\ \n\ ProxyPass / http://localhost:9000/\n\ ProxyPassReverse / http://localhost:9000/\n\ Header add X-Proxied-By "Apache2"' >> /etc/apache2/httpd.conf WORKDIR /app RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT ["/tini", "--", "/app/start.sh"] ``` Run it like this:
{"python": {"version": "3.10.0", "full": "3.10.0 (default, Nov 13 2021, 03:23:03) [GCC 10.3.1 20210424]"}, "datasette": {"version": "0.59.2"}, "asgi": "3.0", "uvicorn": "0.15.0", "sqlite": {"version": "3.35.5", "fts_versions": ["FTS5", "FTS4", "FTS3"], "extensions": {"json1": null}, "compile_options": ["COMPILER=gcc-10.3.1 20210424", "ENABLE_COLUMN_METADATA", "ENABLE_DBSTAT_VTAB", "ENABLE_FTS3", "ENABLE_FTS3_PARENTHESIS", "ENABLE_FTS4", "ENABLE_FTS5", "ENABLE_GEOPOLY", "ENABLE_JSON1", "ENABLE_MATH_FUNCTIONS", "ENABLE_RTREE", "ENABLE_UNLOCK_NOTIFY", "MAX_VARIABLE_NUMBER=250000", "SECURE_DELETE", "THREADSAFE=1", "USE_URI"]}} ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Docker configuration for exercising Datasette behind Apache mod_proxy 1058815557 | |
974371116 | https://github.com/simonw/datasette/issues/1521#issuecomment-974371116 | https://api.github.com/repos/simonw/datasette/issues/1521 | IC_kwDOBm6k_c46E7ks | simonw 9599 | 2021-11-19T19:45:47Z | 2021-11-19T19:45:47Z | OWNER | https://github.com/krallin/tini says:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Docker configuration for exercising Datasette behind Apache mod_proxy 1058815557 | |
974336020 | https://github.com/simonw/datasette/issues/1521#issuecomment-974336020 | https://api.github.com/repos/simonw/datasette/issues/1521 | IC_kwDOBm6k_c46EzAU | simonw 9599 | 2021-11-19T19:10:48Z | 2021-11-19T19:10:48Z | OWNER | There's a promising looking minimal Apache 2 proxy config here: https://stackoverflow.com/questions/26474476/minimal-configuration-for-apache-reverse-proxy-in-docker-container |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Docker configuration for exercising Datasette behind Apache mod_proxy 1058815557 | |
974334278 | https://github.com/simonw/datasette/issues/1521#issuecomment-974334278 | https://api.github.com/repos/simonw/datasette/issues/1521 | IC_kwDOBm6k_c46EylG | simonw 9599 | 2021-11-19T19:08:09Z | 2021-11-19T19:08:09Z | OWNER | Stripping comments using this StackOverflow recipe: https://unix.stackexchange.com/a/157619
Result is here: https://gist.github.com/simonw/0a05090df5fcff8e8b3334621fa17976 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Docker configuration for exercising Datasette behind Apache mod_proxy 1058815557 | |
974332787 | https://github.com/simonw/datasette/issues/1521#issuecomment-974332787 | https://api.github.com/repos/simonw/datasette/issues/1521 | IC_kwDOBm6k_c46EyNz | simonw 9599 | 2021-11-19T19:05:52Z | 2021-11-19T19:05:52Z | OWNER | Made myself this Dockerfile to let me explore a bit: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 CMD ["sh"]
This is the main Apache HTTP server configuration file. It contains theconfiguration directives that give the server its instructions....
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Docker configuration for exercising Datasette behind Apache mod_proxy 1058815557 | |
974327812 | https://github.com/simonw/datasette/issues/1521#issuecomment-974327812 | https://api.github.com/repos/simonw/datasette/issues/1521 | IC_kwDOBm6k_c46ExAE | simonw 9599 | 2021-11-19T18:58:49Z | 2021-11-19T18:59:55Z | OWNER | From this example: https://github.com/tigelane/dockerfiles/blob/06cff2ac8cdc920ebd64f50965115eaa3d0afb84/Alpine-Apache2/Dockerfile#L25-L31 it looks like running ``` APACHE - AlpineRUN apk --update add apache2 php5-apache2 && \ #apk add openrc --no-cache && \ rm -rf /var/cache/apk/* && \ sed -i 's/#ServerName www.example.com:80/ServerName localhost/' /etc/apache2/httpd.conf && \ mkdir -p /run/apache2/ Upload our files from folder "dist".COPY dist /var/www/localhost/htdocs Manually set up the apache environment variablesENV APACHE_RUN_USER www-data ENV APACHE_RUN_GROUP www-data ENV APACHE_LOG_DIR /var/log/apache2 ENV APACHE_LOCK_DIR /var/lock/apache2 ENV APACHE_PID_FILE /var/run/apache2.pid Execute apache2 on runEXPOSE 80 ENTRYPOINT ["httpd"] CMD ["-D", "FOREGROUND"] ``` I think I'll create my own separate copy and modify that. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Docker configuration for exercising Datasette behind Apache mod_proxy 1058815557 | |
974321391 | https://github.com/simonw/datasette/issues/1521#issuecomment-974321391 | https://api.github.com/repos/simonw/datasette/issues/1521 | IC_kwDOBm6k_c46Evbv | simonw 9599 | 2021-11-19T18:49:15Z | 2021-11-19T18:57:18Z | OWNER | This pattern looks like it can help: https://ahmet.im/blog/cloud-run-multiple-processes-easy-way/ - see example in https://github.com/ahmetb/multi-process-container-lazy-solution I got that demo working locally like this:
I want to use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Docker configuration for exercising Datasette behind Apache mod_proxy 1058815557 | |
974322178 | https://github.com/simonw/datasette/issues/1521#issuecomment-974322178 | https://api.github.com/repos/simonw/datasette/issues/1521 | IC_kwDOBm6k_c46EvoC | simonw 9599 | 2021-11-19T18:50:22Z | 2021-11-19T18:50:22Z | OWNER | I'll get this working on my laptop first, but then I want to get it up and running on Cloud Run - maybe with a GitHub Actions workflow in this repo that re-deploys it on manual execution. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Docker configuration for exercising Datasette behind Apache mod_proxy 1058815557 | |
974310208 | https://github.com/simonw/datasette/issues/1519#issuecomment-974310208 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46EstA | simonw 9599 | 2021-11-19T18:32:31Z | 2021-11-19T18:32:31Z | OWNER | Having a live demo running on Cloud Run that proxies through Apache and uses |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974309591 | https://github.com/simonw/datasette/issues/1519#issuecomment-974309591 | https://api.github.com/repos/simonw/datasette/issues/1519 | IC_kwDOBm6k_c46EsjX | simonw 9599 | 2021-11-19T18:31:32Z | 2021-11-19T18:31:32Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
base_url is omitted in JSON and CSV views 1058790545 | |
974308215 | https://github.com/simonw/datasette/issues/1520#issuecomment-974308215 | https://api.github.com/repos/simonw/datasette/issues/1520 | IC_kwDOBm6k_c46EsN3 | simonw 9599 | 2021-11-19T18:29:26Z | 2021-11-19T18:29:26Z | OWNER | The solution that jumps to mind first is that it would be neat if routes could return something that meant "actually my bad, I can't handle this after all - move to the next one in the list". A related idea: it might be useful for custom views like my one here to say "no actually call the default view for this, but give me back the response so I can modify it in some way". Kind of like Django or ASGI middleware. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Pattern for avoiding accidental URL over-rides 1058803238 | |
974300823 | https://github.com/simonw/datasette/issues/1518#issuecomment-974300823 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46EqaX | simonw 9599 | 2021-11-19T18:18:32Z | 2021-11-19T18:18:32Z | OWNER |
I can definitely support this using pure-JSON - I could make two versions of the row available, one that's an array of cell objects and the other that's an object mapping column names to column raw values. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
974285803 | https://github.com/simonw/datasette/issues/1518#issuecomment-974285803 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46Emvr | simonw 9599 | 2021-11-19T17:56:48Z | 2021-11-19T18:14:30Z | OWNER | Very confused by this piece of code here: https://github.com/simonw/datasette/blob/1c13e1af0664a4dfb1e69714c56523279cae09e4/datasette/views/table.py#L37-L63 I added it in https://github.com/simonw/datasette/commit/754836eef043676e84626c4fd3cb993eed0d2976 - in the new world that should probably be replaced by pure JSON. Aha - this comment explains it: https://github.com/simonw/datasette/issues/521#issuecomment-505279560
The goal was to support neater custom templates like this:
```html+jinja
{% for row in display_rows %}
{{ row["First_Name"] }} {{ row["Last_Name"] }}... ``` This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
974287570 | https://github.com/simonw/datasette/issues/1518#issuecomment-974287570 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46EnLS | simonw 9599 | 2021-11-19T17:59:33Z | 2021-11-19T17:59:33Z | OWNER | I'm going to try leaning into the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
974108455 | https://github.com/simonw/datasette/pull/1495#issuecomment-974108455 | https://api.github.com/repos/simonw/datasette/issues/1495 | IC_kwDOBm6k_c46D7cn | mroswell 192568 | 2021-11-19T14:14:35Z | 2021-11-19T14:14:35Z | CONTRIBUTOR | A nudge on this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Allow routes to have extra options 1033678984 | |
973820125 | https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973820125 | https://api.github.com/repos/simonw/sqlite-utils/issues/342 | IC_kwDOCGYnMM46C1Dd | simonw 9599 | 2021-11-19T07:25:55Z | 2021-11-19T07:25:55Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extra options to `lookup()` which get passed to `insert()` 1058196641 | |
973802998 | https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802998 | https://api.github.com/repos/simonw/sqlite-utils/issues/342 | IC_kwDOCGYnMM46Cw32 | simonw 9599 | 2021-11-19T06:59:22Z | 2021-11-19T06:59:32Z | OWNER | I don't think I need the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extra options to `lookup()` which get passed to `insert()` 1058196641 | |
973802766 | https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802766 | https://api.github.com/repos/simonw/sqlite-utils/issues/342 | IC_kwDOCGYnMM46Cw0O | simonw 9599 | 2021-11-19T06:58:45Z | 2021-11-19T06:58:45Z | OWNER | And neither does |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extra options to `lookup()` which get passed to `insert()` 1058196641 | |
973802469 | https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802469 | https://api.github.com/repos/simonw/sqlite-utils/issues/342 | IC_kwDOCGYnMM46Cwvl | simonw 9599 | 2021-11-19T06:58:03Z | 2021-11-19T06:58:03Z | OWNER | Also: I don't think |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extra options to `lookup()` which get passed to `insert()` 1058196641 | |
973802308 | https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802308 | https://api.github.com/repos/simonw/sqlite-utils/issues/342 | IC_kwDOCGYnMM46CwtE | simonw 9599 | 2021-11-19T06:57:37Z | 2021-11-19T06:57:37Z | OWNER | Here's the current full method signature for I could add a test which uses introspection ( |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extra options to `lookup()` which get passed to `insert()` 1058196641 | |
973801650 | https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973801650 | https://api.github.com/repos/simonw/sqlite-utils/issues/342 | IC_kwDOCGYnMM46Cwiy | simonw 9599 | 2021-11-19T06:55:56Z | 2021-11-19T06:55:56Z | OWNER |
I do worry that I'll add more keyword arguments to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extra options to `lookup()` which get passed to `insert()` 1058196641 | |
973800795 | https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973800795 | https://api.github.com/repos/simonw/sqlite-utils/issues/342 | IC_kwDOCGYnMM46CwVb | simonw 9599 | 2021-11-19T06:54:08Z | 2021-11-19T06:54:08Z | OWNER | Looking at the code for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extra options to `lookup()` which get passed to `insert()` 1058196641 | |
973700549 | https://github.com/simonw/datasette/issues/1518#issuecomment-973700549 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46CX3F | simonw 9599 | 2021-11-19T03:31:20Z | 2021-11-19T03:31:26Z | OWNER | ... and while I'm doing all of this I can rewrite the templates to not use those cheating magical functions AND document the template context at the same time, refs: - #1510. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
973700322 | https://github.com/simonw/datasette/issues/1518#issuecomment-973700322 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46CXzi | simonw 9599 | 2021-11-19T03:30:30Z | 2021-11-19T03:30:30Z | OWNER | Right now the HTML version gets to cheat - it passes through objects that are not JSON serializable, including custom functions that can then be called by Jinja. I'm interested in maybe removing this cheating - if the HTML version could only request JSON-serializable extras those could be exposed in the API as well. It would also help cleanup the kind-of-nasty pattern I use in the current |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
973698917 | https://github.com/simonw/datasette/issues/1518#issuecomment-973698917 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46CXdl | simonw 9599 | 2021-11-19T03:26:18Z | 2021-11-19T03:29:03Z | OWNER | A (likely incomplete) list of features on the table page:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
973699424 | https://github.com/simonw/datasette/issues/1518#issuecomment-973699424 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46CXlg | simonw 9599 | 2021-11-19T03:27:49Z | 2021-11-19T03:27:49Z | OWNER | My goal is to break up a lot of this functionality into separate methods. These methods can be executed in parallel by So the HTML version itself needs to be re-written to use those JSON extras. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
973696604 | https://github.com/simonw/datasette/issues/1517#issuecomment-973696604 | https://api.github.com/repos/simonw/datasette/issues/1517 | IC_kwDOBm6k_c46CW5c | simonw 9599 | 2021-11-19T03:20:00Z | 2021-11-19T03:20:00Z | OWNER | Confirmed - my test plugin is indeed correctly over-riding the table page. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Let `register_routes()` over-ride default routes within Datasette 1057996111 | |
973687978 | https://github.com/simonw/datasette/issues/1518#issuecomment-973687978 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46CUyq | simonw 9599 | 2021-11-19T03:07:47Z | 2021-11-19T03:07:47Z | OWNER | I was wrong about that, you CAN over-ride default routes already. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
973686874 | https://github.com/simonw/datasette/issues/1517#issuecomment-973686874 | https://api.github.com/repos/simonw/datasette/issues/1517 | IC_kwDOBm6k_c46CUha | simonw 9599 | 2021-11-19T03:06:58Z | 2021-11-19T03:06:58Z | OWNER | I made a mistake: I just wrote a test that proves that plugins CAN over-ride default routes, plus if you look at the code here the plugins get to register themselves first: https://github.com/simonw/datasette/blob/0156c6b5e52d541e93f0d68e9245f20ae83bc933/datasette/app.py#L965-L981 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Let `register_routes()` over-ride default routes within Datasette 1057996111 | |
973682389 | https://github.com/simonw/datasette/issues/1518#issuecomment-973682389 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46CTbV | simonw 9599 | 2021-11-19T02:57:39Z | 2021-11-19T02:57:39Z | OWNER | Ideally I'd like to execute the existing test suite against the new implementation - that would require me to solve this so I can replace the view with the plugin version though:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
973681970 | https://github.com/simonw/datasette/issues/1518#issuecomment-973681970 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46CTUy | simonw 9599 | 2021-11-19T02:56:31Z | 2021-11-19T02:56:53Z | OWNER | Here's where I got to with my hacked-together initial plugin prototype - it managed to render the table page with some rows on it (and a bunch of missing functionality such as filters): https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
973678931 | https://github.com/simonw/datasette/issues/878#issuecomment-973678931 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46CSlT | simonw 9599 | 2021-11-19T02:51:17Z | 2021-11-19T02:51:17Z | OWNER | OK, I managed to get a table to render! Here's the code I used - I had to copy a LOT of stuff. https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 I'm going to move this work into a new, separate issue. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973635157 | https://github.com/simonw/datasette/issues/878#issuecomment-973635157 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46CH5V | simonw 9599 | 2021-11-19T01:07:08Z | 2021-11-19T01:07:08Z | OWNER | This exercise is proving so useful in getting my head around how the enormous and complex Here's where I've got to now - I'm systematically working through the variables that are returned for HTML and for JSON copying across code to get it to work: ```python from datasette.database import QueryInterrupted from datasette.utils import escape_sqlite from datasette.utils.asgi import Response, NotFound, Forbidden from datasette.views.base import DatasetteError from datasette import hookimpl from asyncinject import AsyncInject, inject from pprint import pformat class Table(AsyncInject): @inject async def database(self, request, datasette): # TODO: all that nasty hash resolving stuff can go here db_name = request.url_vars["db_name"] try: db = datasette.databases[db_name] except KeyError: raise NotFound(f"Database '{db_name}' does not exist") return db
@hookimpl def register_routes(): return [ (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view), ] async def check_permissions(datasette, request, permissions): """permissions is a list of (action, resource) tuples or 'action' strings""" for permission in permissions: if isinstance(permission, str): action = permission resource = None elif isinstance(permission, (tuple, list)) and len(permission) == 2: action, resource = permission else: assert ( False ), "permission should be string or tuple of two items: {}".format( repr(permission) ) ok = await datasette.permission_allowed( request.actor, action, resource=resource, default=None, ) if ok is not None: if ok: return else: raise Forbidden(action) async def columns_to_select(datasette, database, table, request): table_columns = await database.table_columns(table) pks = await database.primary_keys(table) columns = list(table_columns) if "_col" in request.args: columns = list(pks) _cols = request.args.getlist("_col") bad_columns = [column for column in _cols if column not in table_columns] if bad_columns: raise DatasetteError( "_col={} - invalid columns".format(", ".join(bad_columns)), status=400, ) # De-duplicate maintaining order: columns.extend(dict.fromkeys(_cols)) if "_nocol" in request.args: # Return all columns EXCEPT these bad_columns = [ column for column in request.args.getlist("_nocol") if (column not in table_columns) or (column in pks) ] if bad_columns: raise DatasetteError( "_nocol={} - invalid columns".format(", ".join(bad_columns)), status=400, ) tmp_columns = [ column for column in columns if column not in request.args.getlist("_nocol") ] columns = tmp_columns return columns ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973568285 | https://github.com/simonw/datasette/issues/878#issuecomment-973568285 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46B3kd | simonw 9599 | 2021-11-19T00:29:20Z | 2021-11-19T00:29:20Z | OWNER | This is working! ```python from datasette.utils.asgi import Response from datasette import hookimpl import html from asyncinject import AsyncInject, inject class Table(AsyncInject): @inject async def database(self, request): return request.url_vars["db_name"]
@hookimpl
def register_routes():
return [
(r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view),
]
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973564260 | https://github.com/simonw/datasette/issues/878#issuecomment-973564260 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46B2lk | simonw 9599 | 2021-11-19T00:27:06Z | 2021-11-19T00:27:06Z | OWNER | Problem: the fancy @hookimpl def register_routes(): return [ (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view), ] ``` This failed with error: "Table.view() takes 1 positional argument but 2 were given" So I'm going to use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973554024 | https://github.com/simonw/datasette/issues/878#issuecomment-973554024 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46B0Fo | simonw 9599 | 2021-11-19T00:21:20Z | 2021-11-19T00:21:20Z | OWNER | That's annoying: it looks like plugins can't use async def table(request): return Response.html("Hello from {}".format( html.escape(repr(request.url_vars)) )) @hookimpl
def register_routes():
return [
(r"/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", table),
]
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973542284 | https://github.com/simonw/datasette/issues/878#issuecomment-973542284 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46BxOM | simonw 9599 | 2021-11-19T00:16:44Z | 2021-11-19T00:16:44Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973527870 | https://github.com/simonw/datasette/issues/878#issuecomment-973527870 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46Bts- | simonw 9599 | 2021-11-19T00:13:43Z | 2021-11-19T00:13:43Z | OWNER | New plan: I'm going to build a brand new implementation of It will reuse the existing HTML template but will be a completely new Python implementation, based on I'm going to start by just getting the table to show up on the page - then I'll add faceting, suggested facets, filters and so-on. Bonus: I'm going to see if I can get it to work for arbitrary SQL queries too (stretch goal). |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
972858458 | https://github.com/simonw/datasette/pull/1516#issuecomment-972858458 | https://api.github.com/repos/simonw/datasette/issues/1516 | IC_kwDOBm6k_c45_KRa | codecov[bot] 22429695 | 2021-11-18T13:19:01Z | 2021-11-18T13:19:01Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1516 +/-=======================================
Coverage 91.82% 91.82% Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump black from 21.9b0 to 21.11b1 1057340779 | |
972852184 | https://github.com/simonw/datasette/pull/1514#issuecomment-972852184 | https://api.github.com/repos/simonw/datasette/issues/1514 | IC_kwDOBm6k_c45_IvY | dependabot[bot] 49699333 | 2021-11-18T13:11:15Z | 2021-11-18T13:11:15Z | CONTRIBUTOR | Superseded by #1516. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump black from 21.9b0 to 21.11b0 1056117435 | |
971575746 | https://github.com/simonw/datasette/pull/1514#issuecomment-971575746 | https://api.github.com/repos/simonw/datasette/issues/1514 | IC_kwDOBm6k_c456RHC | codecov[bot] 22429695 | 2021-11-17T13:18:58Z | 2021-11-17T13:18:58Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1514 +/-=======================================
Coverage 91.82% 91.82% Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump black from 21.9b0 to 21.11b0 1056117435 | |
971568829 | https://github.com/simonw/datasette/pull/1500#issuecomment-971568829 | https://api.github.com/repos/simonw/datasette/issues/1500 | IC_kwDOBm6k_c456Pa9 | dependabot[bot] 49699333 | 2021-11-17T13:13:58Z | 2021-11-17T13:13:58Z | CONTRIBUTOR | Superseded by #1514. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump black from 21.9b0 to 21.10b0 1041158024 | |
971209475 | https://github.com/simonw/datasette/issues/878#issuecomment-971209475 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4543sD | simonw 9599 | 2021-11-17T05:41:42Z | 2021-11-17T05:41:42Z | OWNER | I'm going to build a brand new implementation of the I can maybe even run the tests against old |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
971057553 | https://github.com/simonw/datasette/issues/878#issuecomment-971057553 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c454SmR | simonw 9599 | 2021-11-17T01:40:45Z | 2021-11-17T01:40:45Z | OWNER | I shipped that code as a new library, |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
971056169 | https://github.com/simonw/datasette/pull/1512#issuecomment-971056169 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c454SQp | simonw 9599 | 2021-11-17T01:39:44Z | 2021-11-17T01:39:44Z | OWNER | Closing this PR because I shipped the code in it as a separate library instead. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
971055677 | https://github.com/simonw/datasette/pull/1512#issuecomment-971055677 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c454SI9 | simonw 9599 | 2021-11-17T01:39:25Z | 2021-11-17T01:39:25Z | OWNER | https://github.com/simonw/asyncinject version 0.1a0 is now live on PyPI: https://pypi.org/project/asyncinject/ |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
971010724 | https://github.com/simonw/datasette/pull/1512#issuecomment-971010724 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c454HKk | simonw 9599 | 2021-11-17T01:12:22Z | 2021-11-17T01:12:22Z | OWNER | I'm going to extract out the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970718652 | https://github.com/simonw/datasette/pull/1512#issuecomment-970718652 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c452_28 | codecov[bot] 22429695 | 2021-11-16T22:02:59Z | 2021-11-16T23:51:48Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1512 +/-==========================================
- Coverage 91.82% 89.72% -2.11% | Impacted Files | Coverage Δ | |
|---|---|---|
| datasette/utils/vendored_graphlib.py | Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970861628 | https://github.com/simonw/datasette/pull/1512#issuecomment-970861628 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c453iw8 | simonw 9599 | 2021-11-16T23:46:07Z | 2021-11-16T23:46:07Z | OWNER | I made the changes locally and tested them with Python 3.6 like so:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970857411 | https://github.com/simonw/datasette/pull/1512#issuecomment-970857411 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c453hvD | simonw 9599 | 2021-11-16T23:43:21Z | 2021-11-16T23:43:21Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970855084 | https://github.com/simonw/datasette/issues/1513#issuecomment-970855084 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453hKs | simonw 9599 | 2021-11-16T23:41:46Z | 2021-11-16T23:41:46Z | OWNER | Conclusion: using a giant convoluted CTE and UNION ALL query to attempt to calculate facets at the same time as retrieving rows is a net LOSS for performance! Very surprised to see that. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970853917 | https://github.com/simonw/datasette/issues/1513#issuecomment-970853917 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453g4d | simonw 9599 | 2021-11-16T23:41:01Z | 2021-11-16T23:41:01Z | OWNER | One very interesting difference between the two: on the single giant query page:
The single big query takes 376ms total to render the page, spending 370ms in 5 queries Those 5 queries, if you're interested```sql select database_name, schema_version from databases PRAGMA schema_version PRAGMA schema_version explain with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet ```All of that additional non-SQL overhead must be stuff relating to Python and template rendering code running on the page. I'm really surprised at how much overhead that is! This is worth researching separately. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970845844 | https://github.com/simonw/datasette/issues/1513#issuecomment-970845844 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453e6U | simonw 9599 | 2021-11-16T23:35:38Z | 2021-11-16T23:35:38Z | OWNER | I tried adding
Compared to: Which is 353ms total. The separate queries ran faster! Really surprising result there. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970828568 | https://github.com/simonw/datasette/issues/1513#issuecomment-970828568 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453asY | simonw 9599 | 2021-11-16T23:27:11Z | 2021-11-16T23:27:11Z | OWNER | One last experiment: I'm going to try running an expensive query in the CTE portion. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970827674 | https://github.com/simonw/datasette/issues/1513#issuecomment-970827674 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453aea | simonw 9599 | 2021-11-16T23:26:58Z | 2021-11-16T23:26:58Z | OWNER | With trace. https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&_trace=1 shows the following:
It didn't run a count because that's the homepage and the count is cached. So I dropped the count from the query and ran it: https://covid-19.datasettes.com/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count()+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count()+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count()+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect++from+truncated%0D%0Aunion+all+select++from+state_facet%0D%0Aunion+all+select++from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1 Shows 649.4359889999259 ms for the query - compared to 755.78843400001ms for the separate. So it saved about 100ms. Still not a huge difference though! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970780866 | https://github.com/simonw/datasette/issues/1513#issuecomment-970780866 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453PDC | simonw 9599 | 2021-11-16T23:01:57Z | 2021-11-16T23:01:57Z | OWNER | One disadvantage to this approach: if you have a SQL time limit of 1s and it takes 0.9s to return the rows but then 0.5s to calculate each of the requested facets the entire query will exceed the time limit. Could work around this by catching that error and then re-running the query just for the rows, but that would result in the user having to wait longer for the results. Could try to remember if that has happened using an in-memory Python data structure and skip the faceting optimization if it's caused problems in the past? That seems a bit gross. Maybe this becomes an opt-in optimization you can request in your What if we kept the query that returns the rows to be displayed on the page separate from the facets, but then executed all of the facets together using this method such that the Maybe a better optimization would be to move facets to happening via |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970766486 | https://github.com/simonw/datasette/issues/1513#issuecomment-970766486 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453LiW | simonw 9599 | 2021-11-16T22:52:56Z | 2021-11-16T22:56:07Z | OWNER | https://covid-19.datasettes.com/covid is 805.2MB https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2 Equivalent SQL: https://covid-19.datasettes.com/covid?sql=with+cte+as+%28%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A%29%2C%0D%0Astate_facet+as+%28%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Afips_facet+as+%28%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Acounty_facet+as+%28%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect++from+truncated%0D%0Aunion+all+select++from+state_facet%0D%0Aunion+all+select++from+fips_facet%0D%0Aunion+all+select++from+county_facet%0D%0Aunion+all+select+*+from+total_count
_facet | facet_name | facet_count | rowid | date | county | state | fips | cases | deaths -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | | | 1917344 | 2021-11-15 | Autauga | Alabama | 1001 | 10407 | 154 | | | 1917345 | 2021-11-15 | Baldwin | Alabama | 1003 | 37875 | 581 | | | 1917346 | 2021-11-15 | Barbour | Alabama | 1005 | 3648 | 79 | | | 1917347 | 2021-11-15 | Bibb | Alabama | 1007 | 4317 | 92 state | Texas | 148028 | | | | | | | state | Georgia | 96249 | | | | | | | state | Virginia | 79315 | | | | | | | fips | | 17580 | | | | | | | fips | 53061 | 665 | | | | | | | fips | 17031 | 662 | | | | | | | county | Washington | 18666 | | | | | | | county | Unknown | 15840 | | | | | | | county | Jefferson | 15637 | | | | | | | COUNT | | 1920593 | | | | | | | |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970770304 | https://github.com/simonw/datasette/issues/1513#issuecomment-970770304 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453MeA | simonw 9599 | 2021-11-16T22:55:19Z | 2021-11-16T22:55:19Z | OWNER | (One thing I really like about this pattern is that it should work exactly the same when used to facet the results of arbitrary SQL queries as it does when faceting results from the table page.) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970767952 | https://github.com/simonw/datasette/issues/1513#issuecomment-970767952 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453L5Q | simonw 9599 | 2021-11-16T22:53:52Z | 2021-11-16T22:53:52Z | OWNER | It's going to take another 15 minutes for the build to finish and deploy the version with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970758179 | https://github.com/simonw/datasette/issues/1513#issuecomment-970758179 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453Jgj | simonw 9599 | 2021-11-16T22:47:38Z | 2021-11-16T22:47:38Z | OWNER | Trace now enabled: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1&_trace=1 Here are the relevant traces:
I modified the query to include the total count as well: https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect++from+truncated%0D%0Aunion+all+select++from+country_long_facet%0D%0Aunion+all+select++from+owner_facet%0D%0Aunion+all+select++from+primary_fuel_facet%0D%0Aunion+all+select+*+from+total_count&_trace=1
To my huge surprise, this convoluted optimization only shaves the sum query time down from 37.8ms to 34.8ms! That entire database file is just 11.1 MB though. Maybe it would make a meaningful difference on something larger? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970742415 | https://github.com/simonw/datasette/issues/1513#issuecomment-970742415 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453FqP | simonw 9599 | 2021-11-16T22:37:14Z | 2021-11-16T22:37:14Z | OWNER | The query takes 42.794ms to run. Here's the equivalent page using separate queries: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1 Annoyingly I can't disable facet suggestions but keep facets. I'm going to turn on tracing so I can see how long the separate queries took. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970738130 | https://github.com/simonw/datasette/issues/1513#issuecomment-970738130 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453EnS | simonw 9599 | 2021-11-16T22:32:19Z | 2021-11-16T22:32:19Z | OWNER | I came up with the following query which seems to work!
Results look like this: _facet | facet_name | facet_count | rowid | country | country_long | name | owner | primary_fuel -- | -- | -- | -- | -- | -- | -- | -- | -- | | | 1 | AFG | Afghanistan | Kajaki Hydroelectric Power Plant Afghanistan | | Hydro | | | 2 | AFG | Afghanistan | Kandahar DOG | | Solar | | | 3 | AFG | Afghanistan | Kandahar JOL | | Solar | | | 4 | AFG | Afghanistan | Mahipar Hydroelectric Power Plant Afghanistan | | Hydro country_long | United States of America | 8688 | | | | | | country_long | China | 4235 | | | | | | country_long | United Kingdom | 2603 | | | | | | owner | | 14112 | | | | | | owner | Lightsource Renewable Energy | 120 | | | | | | owner | Cypress Creek Renewables | 109 | | | | | | primary_fuel | Solar | 9662 | | | | | | primary_fuel | Hydro | 7155 | | | | | | primary_fuel | Wind | 5188 | | | | | | This is a neat proof of concept. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970718337 | https://github.com/simonw/datasette/pull/1512#issuecomment-970718337 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c452_yB | simonw 9599 | 2021-11-16T22:02:30Z | 2021-11-16T22:02:30Z | OWNER | I've decided to make the clever class Simple(AsyncBase): def init(self): self.log = []
class Complex(AsyncBase): inject_all = True
``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970712713 | https://github.com/simonw/datasette/issues/878#issuecomment-970712713 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452-aJ | simonw 9599 | 2021-11-16T21:54:33Z | 2021-11-16T21:54:33Z | OWNER | I'm going to continue working on this in a PR. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970705738 | https://github.com/simonw/datasette/issues/878#issuecomment-970705738 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4528tK | simonw 9599 | 2021-11-16T21:44:31Z | 2021-11-16T21:44:31Z | OWNER | Wrote a TIL about what I learned using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970673085 | https://github.com/simonw/datasette/issues/878#issuecomment-970673085 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4520u9 | simonw 9599 | 2021-11-16T20:58:24Z | 2021-11-16T20:58:24Z | OWNER | New test: ```python class Complex(AsyncBase): def init(self): self.log = []
@pytest.mark.asyncio
async def test_complex():
result = await Complex().go()
# 'c' should only be called once
assert tuple(result) in (
# c and d could happen in either order
("c", "d", "b", "a", "go"),
("d", "c", "b", "a", "go"),
)
try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def new(cls, name, bases, attrs): # Decorate any items that are 'async def' methods registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != "self" and not p.startswith("") } for key, method in _registry.items() } new_attrs["_graph"] = graph return super().new(cls, name, bases, new_attrs) def make_method(method): parameters = inspect.signature(method).parameters.keys()
class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print("\n resolve: ", names) if results is None: results = {}
``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970660299 | https://github.com/simonw/datasette/issues/878#issuecomment-970660299 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452xnL | simonw 9599 | 2021-11-16T20:39:43Z | 2021-11-16T20:42:27Z | OWNER | But that does seem to be the plan that ts = TopologicalSorter(graph)
ts.prepare()
while ts.is_active():
nodes = ts.get_ready()
print(nodes)
ts.done(*nodes)
ts = TopologicalSorter(graph)
ts.prepare()
while ts.is_active():
nodes = ts.get_ready()
print(nodes)
ts.done(nodes)
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970657874 | https://github.com/simonw/datasette/issues/878#issuecomment-970657874 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452xBS | simonw 9599 | 2021-11-16T20:36:01Z | 2021-11-16T20:36:01Z | OWNER | My goal here is to calculate the most efficient way to resolve the different nodes, running them in parallel where possible. So for this class: ```python class Complex(AsyncBase): async def d(self): pass
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970655927 | https://github.com/simonw/datasette/issues/878#issuecomment-970655927 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452wi3 | simonw 9599 | 2021-11-16T20:33:11Z | 2021-11-16T20:33:11Z | OWNER | What should be happening here instead is it should resolve the full graph and notice that So maybe the algorithm I'm inheriting from https://docs.python.org/3/library/graphlib.html isn't the correct algorithm? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970655304 | https://github.com/simonw/datasette/issues/878#issuecomment-970655304 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452wZI | simonw 9599 | 2021-11-16T20:32:16Z | 2021-11-16T20:32:16Z | OWNER | This code is really fiddly. I just got to this version: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def new(cls, name, bases, attrs): # Decorate any items that are 'async def' methods registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != "self" and not p.startswith("") } for key, method in _registry.items() } new_attrs["_graph"] = graph return super().new(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, _results=None, kwargs): print("inner - _results=", _results) parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != "self"] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), "The following DI parameters could not be found in the registry: {}".format( missing ) results = {} results.update(kwargs) if to_resolve: resolved_parameters = await self.resolve(to_resolve, _results) results.update(resolved_parameters) return_value = await method(self, results) if _results is not None: _results[method.name] = return_value return return_value
class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print("\n resolve: ", names) if results is None: results = {}
@pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once assert result == ["c", "b", "a", "go"] ``` This test sometimes passes, and sometimes fails! Output for a pass: ``` tests/test_asyncdi.py inner - _results= None resolve: ['a'] ts.get_ready() returned nodes: ('c', 'b') resolve_nodes ('c', 'b') (current results = {}) awaitables: [<coroutine object Complex.c at 0x1074ac890>, <coroutine object Complex.b at 0x1074ac820>] inner - _results= {} LOG: c inner - _results= {'c': None} resolve: ['c']
ts.get_ready() returned nodes: ('c',)
resolve_nodes ('c',)
(current results = {'c': None})
awaitables: []
End of resolve(), returning {'c': None}
LOG: b
ts.get_ready() returned nodes: ('a',)
resolve_nodes ('a',)
(current results = {'c': None, 'b': None})
awaitables: [<coroutine object Complex.a at 0x1074ac7b0>]
inner - _results= {'c': None, 'b': None}
LOG: a
End of resolve(), returning {'c': None, 'b': None, 'a': None}
LOG: go
resolve: ['a'] ts.get_ready() returned nodes: ('b', 'c') resolve_nodes ('b', 'c') (current results = {}) awaitables: [<coroutine object Complex.b at 0x10923c890>, <coroutine object Complex.c at 0x10923c820>] inner - _results= {} resolve: ['c'] ts.get_ready() returned nodes: ('c',) resolve_nodes ('c',) (current results = {}) awaitables: [<coroutine object Complex.c at 0x10923c6d0>] inner - _results= {} LOG: c inner - _results= {'c': None} LOG: c End of resolve(), returning {'c': None} LOG: b ts.get_ready() returned nodes: ('a',) resolve_nodes ('a',) (current results = {'c': None, 'b': None}) awaitables: [<coroutine object Complex.a at 0x10923c6d0>] inner - _results= {'c': None, 'b': None} LOG: a End of resolve(), returning {'c': None, 'b': None, 'a': None} LOG: go F =================================================================================================== FAILURES =================================================================================================== _______________ test_complex _________________
tests/test_asyncdi.py:48: AssertionError ================== short test summary info ================================ FAILED tests/test_asyncdi.py::test_complex - AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go'] ``` I figured out why this is happening.
The code decides to run If If |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970624197 | https://github.com/simonw/datasette/issues/878#issuecomment-970624197 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452ozF | simonw 9599 | 2021-11-16T19:49:05Z | 2021-11-16T19:49:05Z | OWNER | Here's the latest version of my weird dependency injection async class: ```python import inspect class AsyncMeta(type): def new(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.name == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value
def make_method(method): @wraps(method) async def inner(self, kwargs): parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != "self"] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), "The following DI parameters could not be found in the registry: {}".format( missing ) results = {} results.update(kwargs) results.update(await self.resolve(to_resolve)) return await method(self, results)
bad = [0] class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names): print(" resolve({})".format(names)) results = {} # Resolve them in the correct order ts = TopologicalSorter() ts2 = TopologicalSorter() print(" names = ", names) print(" self._graph = ", self._graph) for name in names: if self._graph[name]: ts.add(name, self._graph[name]) ts2.add(name, self._graph[name]) print(" static_order =", tuple(ts2.static_order())) ts.prepare() while ts.is_active(): print(" is_active, i = ", bad[0]) bad[0] += 1 if bad[0] > 20: print(" Infinite loop?") break nodes = ts.get_ready() print(" Do nodes:", nodes) awaitables = [self._registryname for name in nodes] print(" awaitables: ", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update({ p[0].name: p[1] for p in zip(awaitables, awaitable_results) }) print(results) for node in nodes: ts.done(node)
foo = Foo()
await foo.other()
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970554697 | https://github.com/simonw/datasette/issues/782#issuecomment-970554697 | https://api.github.com/repos/simonw/datasette/issues/782 | IC_kwDOBm6k_c452X1J | simonw 9599 | 2021-11-16T18:32:03Z | 2021-11-16T18:32:03Z | OWNER | I'm going to take another look at this: - https://github.com/simonw/datasette/issues/878 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user >30