{"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-805047117", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 805047117, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNTA0NzExNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-23T16:30:15Z", "updated_at": "2021-03-23T16:46:06Z", "author_association": "OWNER", "body": "https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js is only 12.5KB zipped, 38KB total - so that's not a bad option.\r\n\r\nhttps://github.com/nodeca/js-yaml", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1270#issuecomment-805058241", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1270", "id": 805058241, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNTA1ODI0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-23T16:45:39Z", "updated_at": "2021-03-23T16:45:39Z", "author_association": "OWNER", "body": "I managed to build SpatiaLite such that this isn't necessary any more. I'm still interested in pursuing this further though - it feels like it could be a more robust way of implementing timeouts, but I need to prove to myself that it's better (maybe better performance, or handles more edge-cases?). Not sure how to prove that yet.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837350092, "label": "Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-805056806", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 805056806, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNTA1NjgwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-23T16:43:38Z", "updated_at": "2021-03-23T16:43:38Z", "author_association": "OWNER", "body": "I used this code to get that:\r\n```javascript\r\nvar jsonVersion = JSON.stringify(window.jsyaml.load(document.querySelector('.highlight-yaml').textContent), null, 4);\r\ndiv.querySelector('.highlight pre').innerText = jsonVersion;\r\ndiv.querySelector('.highlight pre').style.whiteSpace = 'pre-wrap'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-805055291", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 805055291, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNTA1NTI5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-23T16:41:31Z", "updated_at": "2021-03-23T16:41:31Z", "author_association": "OWNER", "body": "One downside of doing this conversion in JavaScript: it's much harder to get the same JSON syntax highlighting as that provided by Sphinx:\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-805050163", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 805050163, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNTA1MDE2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-23T16:34:35Z", "updated_at": "2021-03-23T16:35:32Z", "author_association": "OWNER", "body": "https://docs.datasette.io/en/stable/metadata.html has this example:\r\n```yaml\r\ntitle: Demonstrating Metadata from YAML\r\ndescription_html: |-\r\n
This description includes a long HTML string
\r\nThis demonstrates basic LIKE search\r\n```\r\n\r\nI ran this in the browser dev tools:\r\n```javascript\r\nvar s = document.createElement('script')\r\ns.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js'\r\ndocument.head.appendChild(s)\r\nvar yamlExample = document.querySelector('.highlight-yaml').textContent);\r\nconsole.log(JSON.stringify(window.jsyaml.load(yamlExample), null, 4))\r\n```\r\nAnd got:\r\n```json\r\n{\r\n \"title\": \"Demonstrating Metadata from YAML\",\r\n \"description_html\": \"
This description includes a long HTML string
\\nThis demonstrates basic LIKE search\"\r\n }\r\n }\r\n }\r\n }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-805042880", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 805042880, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNTA0Mjg4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-23T16:24:32Z", "updated_at": "2021-03-23T16:24:32Z", "author_association": "OWNER", "body": "... actually I think I would do that conversion in Python. The client-side YAML parsers all look a little bit heavy to me in terms of additional page weight.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-805041522", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 805041522, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNTA0MTUyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-23T16:22:46Z", "updated_at": "2021-03-23T16:22:46Z", "author_association": "OWNER", "body": "That's a good idea. I could do that with JavaScript - loading YAML and converting it to JSON in JavaScript shouldn't be hard, and it's better than JSON-to-YAML because there's only one correct JSON representation of a YAML file whereas you can represent a JSON document in YAML in a bunch of different ways.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-805033155", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 805033155, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNTAzMzE1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-23T16:12:13Z", "updated_at": "2021-03-23T16:12:13Z", "author_association": "OWNER", "body": "Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source\r\n\r\n> The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful [VirtualKNN module](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN)), take a look at the [Datasette Dockerfile](https://github.com/simonw/datasette/blob/master/Dockerfile).\r\n\r\nSee also #1273", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/163#issuecomment-804540869", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/163", "id": 804540869, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDU0MDg2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-23T02:44:33Z", "updated_at": "2021-03-23T02:44:33Z", "author_association": "OWNER", "body": "Comments welcome!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 279547886, "label": "Document the querystring argument for setting a different time limit"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1271#issuecomment-804299406", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1271", "id": 804299406, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDI5OTQwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T18:36:14Z", "updated_at": "2021-03-22T21:49:27Z", "author_association": "OWNER", "body": "This isn't actually working - the outer code attempts to send an `.interrupt()` call to the connection object via the `connections` thread-local, which doesn't work because it's a thread-local so the connection isn't visible to that code.\r\n\r\nNeed to figure out how to communicate with that thread properly.\r\n\r\nAlso a test that fails in this particular case would be a good idea!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837956424, "label": "Use SQLite conn.interrupt() instead of sqlite_timelimit()"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804406675", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804406675, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDQwNjY3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T21:26:27Z", "updated_at": "2021-03-22T21:26:27Z", "author_association": "OWNER", "body": "(Without the `apt-get update ...` SpatiaLite line it's 125MB)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804404544", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804404544, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T21:22:56Z", "updated_at": "2021-03-22T21:24:24Z", "author_association": "OWNER", "body": "Final version of Dockerfile which installs the specified version from GitHub:\r\n\r\n docker build . -t datasette-spatialite --build-arg VERSION=0.55\r\n\r\n```dockerfile\r\nFROM python:3.9.2-slim-buster as build\r\n\r\n# Version of Datasette to install, e.g. 0.55\r\n# docker build . -t datasette --build-arg VERSION=0.55\r\nARG VERSION\r\n\r\n# software-properties-common provides add-apt-repository\r\n# which we need in order to install a more recent release\r\n# of libsqlite3-mod-spatialite from the sid distribution\r\nRUN apt-get update && \\\r\n apt-get -y --no-install-recommends install software-properties-common && \\\r\n add-apt-repository \"deb http://httpredir.debian.org/debian sid main\" && \\\r\n apt-get update && \\\r\n apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \\\r\n apt-get remove -y software-properties-common && \\\r\n apt clean && \\\r\n rm -rf /var/lib/apt && \\\r\n rm -rf /var/lib/dpkg\r\n\r\nRUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \\\r\n find /usr/local/lib -name '__pycache__' | xargs rm -r && \\\r\n rm -rf /root/.cache/pip\r\n\r\nEXPOSE 8001\r\nCMD [\"datasette\"]\r\n```\r\nRun against 0.55 this produces an image of 262MB", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804384196", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804384196, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T20:48:46Z", "updated_at": "2021-03-22T20:48:46Z", "author_association": "OWNER", "body": "I think part of the reason it's smaller is that I ran `pip install datasette` instead of using `COPY . /datasette` followed by `pip install /datasette`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804380181", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804380181, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T20:42:16Z", "updated_at": "2021-03-22T20:42:16Z", "author_association": "OWNER", "body": "Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804379644", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804379644, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T20:41:23Z", "updated_at": "2021-03-22T20:41:23Z", "author_association": "OWNER", "body": "I tried adding `apt-get remove -y software-properties-common &&` to remove `software-properties-common` but it made no difference to the image size.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804372977", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804372977, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T20:30:37Z", "updated_at": "2021-03-22T20:30:37Z", "author_association": "OWNER", "body": "I tried copying just the `mod_spatialite.so` file into a second stage build but it failed. So I ran `bash` in a working image and used `ldd` to figure out what it was linked to:\r\n```\r\nroot@39683f91e588:/usr/lib/x86_64-linux-gnu# ldd mod_spatialite.so\r\n\tlinux-vdso.so.1 (0x00007ffd021f4000)\r\n\tlibxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f5c75412000)\r\n\tlibpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c753f0000)\r\n\tlibm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c752ac000)\r\n\tlibminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f5c750a0000)\r\n\tlibrttopo.so.1 => /usr/lib/x86_64-linux-gnu/librttopo.so.1 (0x00007f5c75028000)\r\n\tlibfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f5c7501c000)\r\n\tlibproj.so.19 => /usr/lib/x86_64-linux-gnu/libproj.so.19 (0x00007f5c74ca7000)\r\n\tlibz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c74a89000)\r\n\tlibsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f5c74967000)\r\n\tlibgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f5c7492b000)\r\n\tlibc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c74766000)\r\n\tlibdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c74760000)\r\n\tlibicuuc.so.67 => /usr/lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007f5c74575000)\r\n\tliblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5c7454d000)\r\n\t/lib64/ld-linux-x86-64.so.2 (0x00007f5c75d49000)\r\n\tlibtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f5c744c7000)\r\n\tlibcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f5c74439000)\r\n\tlibstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5c7426c000)\r\n\tlibgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c74250000)\r\n\tlibgeos-3.9.0.so => /usr/lib/x86_64-linux-gnu/libgeos-3.9.0.so (0x00007f5c74040000)\r\n\tlibicudata.so.67 => /usr/lib/x86_64-linux-gnu/libicudata.so.67 (0x00007f5c72527000)\r\n\tlibwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f5c724bc000)\r\n\tlibzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f5c7241c000)\r\n\tlibjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f5c7220e000)\r\n\tlibjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f5c72188000)\r\n\tlibdeflate.so.0 => /usr/lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f5c7216c000)\r\n\tlibnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f5c72144000)\r\n\tlibidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f5c72125000)\r\n\tlibrtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f5c71f08000)\r\n\tlibssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f5c71eda000)\r\n\tlibpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f5c71ec5000)\r\n\tlibnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f5c71e8d000)\r\n\tlibgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f5c71ce0000)\r\n\tlibgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f5c71c93000)\r\n\tlibkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f5c71bb3000)\r\n\tlibk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f5c71b7f000)\r\n\tlibcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f5c71b77000)\r\n\tlibldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f5c71b23000)\r\n\tliblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f5c71b12000)\r\n\tlibunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f5c7198e000)\r\n\tlibhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f5c71955000)\r\n\tlibgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f5c718d0000)\r\n\tlibgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f5c717b2000)\r\n\tlibp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f5c71683000)\r\n\tlibtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f5c71470000)\r\n\tlibkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f5c71461000)\r\n\tlibkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f5c71458000)\r\n\tlibresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5c7143e000)\r\n\tlibsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f5c71421000)\r\n\tlibgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5c713fe000)\r\n\tlibffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f5c713f4000)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804368372", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804368372, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T20:22:43Z", "updated_at": "2021-03-22T20:22:43Z", "author_association": "OWNER", "body": "```dockerfile\r\nFROM python:3.9.2-slim-buster as build\r\n\r\n# software-properties-common provides add-apt-repository\r\nRUN apt-get update && \\\r\n apt-get -y --no-install-recommends install software-properties-common && \\\r\n add-apt-repository \"deb http://httpredir.debian.org/debian sid main\" && \\\r\n apt-get update && \\\r\n apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \\\r\n apt clean && \\\r\n rm -rf /var/lib/apt && \\\r\n rm -rf /var/lib/dpkg\r\n\r\nRUN pip install datasette && \\\r\n find /usr/local/lib -name '__pycache__' | xargs rm -r && \\\r\n rm -rf /root/.cache/pip\r\n\r\nEXPOSE 8001\r\nCMD [\"datasette\"]\r\n```\r\n262 MB", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804363687", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804363687, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T20:15:00Z", "updated_at": "2021-03-22T20:15:00Z", "author_association": "OWNER", "body": "```\r\nRUN pip install datasette && \\\r\n find /usr/local/lib -name '__pycache__' | xargs rm -r\r\n```\r\nThat dropped it to 265MB.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804360701", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804360701, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDM2MDcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T20:10:07Z", "updated_at": "2021-03-22T20:10:07Z", "author_association": "OWNER", "body": "Adding `--no-install-recommends` dropped it to 275MB", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804347152", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804347152, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T19:47:56Z", "updated_at": "2021-03-22T19:48:03Z", "author_association": "OWNER", "body": "I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804344553", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804344553, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDM0NDU1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T19:43:25Z", "updated_at": "2021-03-22T19:43:25Z", "author_association": "OWNER", "body": "Does `--no-install-recommends` make a difference?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804338678", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804338678, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T19:33:43Z", "updated_at": "2021-03-22T19:33:43Z", "author_association": "OWNER", "body": "Replacing `rm -rf /var/lib/{apt,dpkg,cache,log}/` with \r\n```\r\n rm -rf /var/lib/apt && \\\r\n rm -rf /var/lib/dpkg\r\n```\r\nGot the size down to 305MB.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804318314", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804318314, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDMxODMxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T19:04:30Z", "updated_at": "2021-03-22T19:04:30Z", "author_association": "OWNER", "body": "Considering the image on Docker Hub right now is `383MB` this is actually an improvement.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804317545", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804317545, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T19:03:22Z", "updated_at": "2021-03-22T19:03:22Z", "author_association": "OWNER", "body": "This Dockerfile:\r\n```dockerfile\r\nFROM python:3.9.2-slim-buster as build\r\n\r\n# software-properties-common provides add-apt-repository\r\nRUN apt-get update && \\\r\n apt-get -y install software-properties-common && \\\r\n add-apt-repository \"deb http://httpredir.debian.org/debian sid main\" && \\\r\n apt-get update && \\\r\n apt-get -t sid install -y libsqlite3-mod-spatialite && \\\r\n apt clean && \\\r\n rm -rf /var/lib/{apt,dpkg,cache,log}/\r\n\r\nRUN pip install datasette\r\n\r\nEXPOSE 8001\r\nCMD [\"datasette\"]\r\n```\r\nProduces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804310353", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804310353, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T18:52:12Z", "updated_at": "2021-03-22T18:52:12Z", "author_association": "OWNER", "body": "This Dockerfile:\r\n```dockerfile\r\nFROM python:3.9.2-slim-buster as build\r\n\r\n# Setup build dependencies\r\nRUN apt update \\\r\n && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \\\r\n libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \\\r\n && apt clean\r\n\r\nRUN wget \"https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz\" && tar xzf sqlite-autoconf-3340100.tar.gz \\\r\n && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \\\r\n CFLAGS=\"-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1\" \\\r\n && make && make install\r\n\r\nRUN wget \"http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz\" && tar zxf freexl-1.0.6.tar.gz \\\r\n && cd freexl-1.0.6 && ./configure && make && make install\r\n\r\nRUN wget \"http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz\" && tar zxf libspatialite-5.0.1.tar.gz \\\r\n && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install\r\n\r\nRUN wget \"http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz\" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install\r\n\r\nRUN wget \"http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz\" && tar zxf spatialite-tools-5.0.0.tar.gz \\\r\n && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install\r\n\r\n# Add local code to the image instead of fetching from pypi.\r\n#COPY . /datasette\r\n#RUN pip install /datasette\r\nRUN pip install datasette\r\n\r\nFROM python:3.9.2-slim-buster\r\n\r\n# Copy python dependencies and spatialite libraries\r\nCOPY --from=build /usr/local/lib/ /usr/local/lib/\r\n# Copy executables\r\nCOPY --from=build /usr/local/bin /usr/local/bin\r\n# Copy spatial extensions\r\nCOPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu\r\n\r\nENV LD_LIBRARY_PATH=/usr/local/lib\r\n\r\nEXPOSE 8001\r\nCMD [\"datasette\"]\r\n```\r\nProduced a 448MB image.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804309510", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804309510, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T18:50:50Z", "updated_at": "2021-03-22T18:50:50Z", "author_association": "OWNER", "body": "Ideally I'd like to use the Debian stable `python:3.9.2-slim-buster` base image but install SpatiaLite from Debian unstable here: https://packages.debian.org/sid/libspatialite7\r\n\r\nThis pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1271#issuecomment-804265042", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1271", "id": 804265042, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDI2NTA0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T17:45:45Z", "updated_at": "2021-03-22T17:45:45Z", "author_association": "OWNER", "body": "I can remove this code too: \r\nhttps://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L190-L192", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837956424, "label": "Use SQLite conn.interrupt() instead of sqlite_timelimit()"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-804263434", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 804263434, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T17:43:25Z", "updated_at": "2021-03-22T17:43:25Z", "author_association": "OWNER", "body": "I figured out the cause of the hang in #1268 - it was caused by `select count(*) from SpatialIndex` interacting badly with the `set_progress_handler()` mechanism I was using to implement query time limits. #1271 has a replacement for that using `asyncio.wait_for()` and `conn.interrupt()` which should resolve the SpatiaLite issue too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-804261915", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 804261915, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDI2MTkxNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T17:41:12Z", "updated_at": "2021-03-22T17:41:12Z", "author_association": "OWNER", "body": "Closing this because I've figured out the root of the problem now, and I have a potential solution.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1269#issuecomment-804261610", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1269", "id": 804261610, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDI2MTYxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T17:40:41Z", "updated_at": "2021-03-22T17:40:41Z", "author_association": "OWNER", "body": "#1270 looks promising, and I don't want to leave open a security hole where someone could potentially hang Datasette with a nasty `count(*)` query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837348479, "label": "Don't attempt to run count(*) against virtual tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1270#issuecomment-804255633", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1270", "id": 804255633, "node_id": "MDEyOklzc3VlQ29tbWVudDgwNDI1NTYzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T17:32:02Z", "updated_at": "2021-03-22T17:32:08Z", "author_association": "OWNER", "body": "Confirmed that the `interrupt()` based cancellation mechanism fixes the SpatiaLite issue in #1268!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837350092, "label": "Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler()"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1270#issuecomment-803834784", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1270", "id": 803834784, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzgzNDc4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T07:31:57Z", "updated_at": "2021-03-22T16:22:19Z", "author_association": "OWNER", "body": "I think the implementation for this goes here: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L146-L157\r\n\r\nI figured out a similar pattern in `datasette-ripgrep` here: https://github.com/simonw/datasette-ripgrep/blob/0.7/datasette_ripgrep/__init__.py#L63-L71", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837350092, "label": "Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler()"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803802957", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803802957, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzgwMjk1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T06:38:14Z", "updated_at": "2021-03-22T06:38:14Z", "author_association": "OWNER", "body": "Also worth trying is to change this code:\r\n```python\r\n n = 1000 \r\n if ms < 50: \r\n n = 1 \r\n```\r\nWhat happens with `n = 10` instead?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1269#issuecomment-803785808", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1269", "id": 803785808, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc4NTgwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T06:00:53Z", "updated_at": "2021-03-22T06:00:53Z", "author_association": "OWNER", "body": "This may not be necessary if using `.interrupt() for SQLite timeouts in #1270 works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837348479, "label": "Don't attempt to run count(*) against virtual tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803784902", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803784902, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc4NDkwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:59:06Z", "updated_at": "2021-03-22T05:59:06Z", "author_association": "OWNER", "body": "Even if I implement that workaround in #1269 I'm concerned that this could still allow users to deliberately crash Datasette (if it's running SpatiaLite 5.0) by executing `select count(*) from SpatialIndex`.\r\n\r\nThat `interrupt` timeout mechanism is worth digging into further.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803782705", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803782705, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc4MjcwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:54:19Z", "updated_at": "2021-03-22T05:54:19Z", "author_association": "OWNER", "body": "Got two new TILs out of this:\r\n\r\n* [Tracing every executed Python statement](https://til.simonwillison.net/python/tracing-every-statement)\r\n* [Running gdb against a Python process in a running Docker container](https://til.simonwillison.net/docker/gdb-python-docker)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803777724", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803777724, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc3NzcyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:42:50Z", "updated_at": "2021-03-22T05:43:23Z", "author_association": "OWNER", "body": "\r\n\r\nIf I want to avoid counting virtual tables, I need to detect which tables are virtual tables.\r\n\r\nThe safest way to do this is probably to pull the `sql` for every table and then, in Python, check for values that start with `create virtual table` after converting to lower case, using any number of spaces.\r\n\r\nThis would catch things like ` CREATE virtual TABLE` which might be missed by a SQL `like` query. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803775121", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803775121, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc3NTEyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:36:26Z", "updated_at": "2021-03-22T05:36:26Z", "author_association": "OWNER", "body": "So one fix could be to avoid running counts for anything that turns out to be a virtual table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803774926", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803774926, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc3NDkyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:35:56Z", "updated_at": "2021-03-22T05:35:56Z", "author_association": "OWNER", "body": "That's in this code here: https://github.com/simonw/datasette/blob/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b/datasette/database.py#L221-L241", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803774518", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803774518, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc3NDUxOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:34:57Z", "updated_at": "2021-03-22T05:34:57Z", "author_association": "OWNER", "body": "... and sure enough, adding this code fixed the problem:\r\n```diff\r\ndiff --git a/datasette/database.py b/datasette/database.py\r\nindex 3579cce..b466b12 100644\r\n--- a/datasette/database.py\r\n+++ b/datasette/database.py\r\n@@ -224,6 +226,9 @@ class Database:\r\n # Try to get counts for each table, $limit timeout for each count\r\n counts = {}\r\n for table in await self.table_names():\r\n+ if table == \"SpatialIndex\":\r\n+ counts[table] = 0\r\n+ continue\r\n try:\r\n table_count = (\r\n await self.execute(\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803773484", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803773484, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc3MzQ4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:32:29Z", "updated_at": "2021-03-22T05:32:29Z", "author_association": "OWNER", "body": "To figure out which SQL query triggers the problem I added this code to write to a log file:\r\n```python\r\n with sqlite_timelimit(conn, time_limit_ms):\r\n try:\r\n cursor = conn.cursor()\r\n with open(\"/tmp/sql.log\", \"ab\", buffering=0) as fp:\r\n fp.write((\"{}: {}\\n\".format(sql, params)).encode(\"utf-8\"))\r\n cursor.execute(sql, params if params is not None else {})\r\n```\r\nI had to use `ab` binary mode because Python doesn't allow `buffering=0` for non-binary file operations.\r\n\r\nWith the log enabled, I used `docker exec -it 589ae68de943 bash` to attach to the running container and `tail -f /tmp/sql.log` to see the logs. Here's where it broke:\r\n\r\n```\r\nselect count(*) from [idx_civici_geom_parent]: None\r\nselect count(*) from [sqlite_stat1]: None\r\nselect count(*) from [sqlite_stat3]: None\r\nselect count(*) from [SpatialIndex]: None\r\n```\r\nSo attempting to run a `count(*)` against the `SpatialIndex` virtual table is the thing that triggers the bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803764919", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803764919, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc2NDkxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:11:11Z", "updated_at": "2021-03-22T05:11:11Z", "author_association": "OWNER", "body": "Maybe I could implement SQLite query timeouts using the `interrupt()` method instead of the progress handler hack I'm currently using?\r\n\r\nhttps://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803764200", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803764200, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc2NDIwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:09:13Z", "updated_at": "2021-03-22T05:09:13Z", "author_association": "OWNER", "body": "I tried building a container where the `conn.set_progress_handler(handler, n)` line was commented out... and it fixed the bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803762969", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803762969, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc2Mjk2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:05:51Z", "updated_at": "2021-03-22T05:05:51Z", "author_association": "OWNER", "body": "I had to run `docker kill 16197781a7b5` to kill the broken container - Ctrl+C in the Datasette console window didn't do anything.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 837308703, "label": "Figure out why SpatiaLite 5.0 hangs the database page on Linux"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1268#issuecomment-803762609", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1268", "id": 803762609, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzc2MjYwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T05:05:00Z", "updated_at": "2021-03-22T05:05:00Z", "author_association": "OWNER", "body": "Using https://til.simonwillison.net/docker/attach-bash-to-running-container - I figured out how to run `gdb`. I had to use `--privileged` here because otherwise `gdb` showed a \"Could not attach to process\" error.\r\n```\r\ndocker exec --privileged -it 16197781a7b5 bash\r\n# apt-get install gdb python3-dbg\r\n# gdb /usr/bin/python3 -p 20\r\n```\r\nThis paused the process. I tried running this:\r\n```\r\n(gdb) py-bt\r\nTraceback (most recent call first):\r\n File \"/usr/lib/python3.8/asyncio/base_events.py\", line 1845, in _run_once\r\n if handle._cancelled:\r\n File \"/usr/lib/python3.8/asyncio/base_events.py\", line 570, in run_forever\r\n self._run_once()\r\n File \"/usr/lib/python3.8/asyncio/base_events.py\", line 603, in run_until_complete\r\n self.run_forever()\r\n File \"/usr/local/lib/python3.8/dist-packages/uvicorn/server.py\", line 49, in run\r\n loop.run_until_complete(self.serve(sockets=sockets))\r\n File \"/usr/local/lib/python3.8/dist-packages/uvicorn/main.py\", line 386, in run\r\n server.run()\r\n File \"/usr/local/lib/python3.8/dist-packages/datasette/cli.py\", line 575, in serve\r\n uvicorn.run(ds.app(), **uvicorn_kwargs)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/bin/datasette\", line 8, in