html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/issues/1273#issuecomment-808759984,https://api.github.com/repos/simonw/datasette/issues/1273,808759984,MDEyOklzc3VlQ29tbWVudDgwODc1OTk4NA==,9599,2021-03-27T16:43:17Z,2021-03-27T16:43:17Z,OWNER,That rivers example in the tutorial would work a lot better with a live demo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890, https://github.com/simonw/datasette/issues/1273#issuecomment-808756921,https://api.github.com/repos/simonw/datasette/issues/1273,808756921,MDEyOklzc3VlQ29tbWVudDgwODc1NjkyMQ==,9599,2021-03-27T16:19:45Z,2021-03-27T16:26:28Z,OWNER,"I have a better recipe for using spatial indexes now on https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/ ```sql select AsGeoJSON(geometry), * from CPAD_2020a_SuperUnits where PARK_NAME like '%mini%' and Intersects(GeomFromGeoJSON(:freedraw), geometry) = 1 and CPAD_2020a_SuperUnits.rowid in ( select rowid from SpatialIndex where f_table_name = 'CPAD_2020a_SuperUnits' and search_frame = GeomFromGeoJSON(:freedraw) ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890, https://github.com/simonw/datasette/issues/1273#issuecomment-808757721,https://api.github.com/repos/simonw/datasette/issues/1273,808757721,MDEyOklzc3VlQ29tbWVudDgwODc1NzcyMQ==,9599,2021-03-27T16:25:48Z,2021-03-27T16:25:48Z,OWNER,"> This will give you back an additional column of GeoJSON. You can copy and paste GeoJSON from this column into the debugging tool at geojson.io to visualize it on a map. That should promote `datasette-leaflet-geojson` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890, https://github.com/simonw/datasette/issues/1090#issuecomment-808757659,https://api.github.com/repos/simonw/datasette/issues/1090,808757659,MDEyOklzc3VlQ29tbWVudDgwODc1NzY1OQ==,9599,2021-03-27T16:25:25Z,2021-03-27T16:25:25Z,OWNER,Related feature request: ability to set default values for canned queries: #1258,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364, https://github.com/simonw/datasette/issues/1090#issuecomment-808757155,https://api.github.com/repos/simonw/datasette/issues/1090,808757155,MDEyOklzc3VlQ29tbWVudDgwODc1NzE1NQ==,9599,2021-03-27T16:21:43Z,2021-03-27T16:21:43Z,OWNER,"Idea for these: imitate https://django-sql-dashboard.readthedocs.io/en/latest/widgets.html#custom-widgets and drive them with templates. So a custom widget type of `textarea` would look for a template called `widgets/textarea.html` - which means users could define brand new custom widgets just by creating their own template files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364, https://github.com/simonw/datasette/issues/1278#issuecomment-808756366,https://api.github.com/repos/simonw/datasette/issues/1278,808756366,MDEyOklzc3VlQ29tbWVudDgwODc1NjM2Ng==,9599,2021-03-27T16:15:47Z,2021-03-27T16:15:47Z,OWNER,https://timezones-api.datasette.io/ is now up and running on Cloud Run.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842416110, https://github.com/simonw/datasette/issues/1278#issuecomment-808652008,https://api.github.com/repos/simonw/datasette/issues/1278,808652008,MDEyOklzc3VlQ29tbWVudDgwODY1MjAwOA==,9599,2021-03-27T04:47:17Z,2021-03-27T04:47:17Z,OWNER,"https://github.com/simonw/timezones-api is that project, it's pretty old now. I'll try to get it running on Cloud Run.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842416110, https://github.com/simonw/datasette/issues/1258#issuecomment-808651088,https://api.github.com/repos/simonw/datasette/issues/1258,808651088,MDEyOklzc3VlQ29tbWVudDgwODY1MTA4OA==,9599,2021-03-27T04:41:52Z,2021-03-27T04:42:14Z,OWNER,"Right now they look like this: ```yaml databases: fixtures: queries: neighborhood_search: params: - text ``` In addition to being able to specify defaults, I'd also like to add other things in the future - most significantly the ability to specify a different input widget (e.g. textarea v.s. single-line input) So maybe this looks like: ```yaml params: - name: text default: """" - name: age widget: number ```","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421, https://github.com/simonw/datasette/issues/1258#issuecomment-808650266,https://api.github.com/repos/simonw/datasette/issues/1258,808650266,MDEyOklzc3VlQ29tbWVudDgwODY1MDI2Ng==,9599,2021-03-27T04:37:07Z,2021-03-27T04:37:07Z,OWNER,I like that idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421, https://github.com/simonw/datasette/issues/1249#issuecomment-808649480,https://api.github.com/repos/simonw/datasette/issues/1249,808649480,MDEyOklzc3VlQ29tbWVudDgwODY0OTQ4MA==,9599,2021-03-27T04:32:10Z,2021-03-27T04:32:10Z,OWNER,I'll close this issue after I ship Datasette 0.56 and confirm that the Dockerfile was correctly built and published to Docker Hub.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069, https://github.com/simonw/datasette/issues/1249#issuecomment-808649322,https://api.github.com/repos/simonw/datasette/issues/1249,808649322,MDEyOklzc3VlQ29tbWVudDgwODY0OTMyMg==,9599,2021-03-27T04:31:28Z,2021-03-27T04:31:28Z,OWNER,"One last test of that Dockerfile: ``` (datasette) datasette % docker build -f Dockerfile -t datasetteproject/datasette:0.55a --build-arg VERSION=0.55 . (datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' | jq { ""python"": { ""version"": ""3.9.2"", ""full"": ""3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"" }, ""datasette"": { ""version"": ""0.55"" }, ""asgi"": ""3.0"", ""uvicorn"": ""0.13.4"", ""sqlite"": { ""version"": ""3.27.2"", ""fts_versions"": [ ""FTS5"", ""FTS4"", ""FTS3"" ], ""extensions"": { ""json1"": null }, ""compile_options"": [ ""COMPILER=gcc-8.3.0"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_DBSTAT_VTAB"", ""ENABLE_FTS3"", ""ENABLE_FTS3_PARENTHESIS"", ""ENABLE_FTS3_TOKENIZER"", ""ENABLE_FTS4"", ""ENABLE_FTS5"", ""ENABLE_JSON1"", ""ENABLE_LOAD_EXTENSION"", ""ENABLE_PREUPDATE_HOOK"", ""ENABLE_RTREE"", ""ENABLE_SESSION"", ""ENABLE_STMTVTAB"", ""ENABLE_UNLOCK_NOTIFY"", ""ENABLE_UPDATE_DELETE_LIMIT"", ""HAVE_ISNAN"", ""LIKE_DOESNT_MATCH_BLOBS"", ""MAX_SCHEMA_RETRY=25"", ""MAX_VARIABLE_NUMBER=250000"", ""OMIT_LOOKASIDE"", ""SECURE_DELETE"", ""SOUNDEX"", ""TEMP_STORE=1"", ""THREADSAFE=1"", ""USE_URI"" ] } } (datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' --load-extension=spatialite | jq { ""python"": { ""version"": ""3.9.2"", ""full"": ""3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"" }, ""datasette"": { ""version"": ""0.55"" }, ""asgi"": ""3.0"", ""uvicorn"": ""0.13.4"", ""sqlite"": { ""version"": ""3.27.2"", ""fts_versions"": [ ""FTS5"", ""FTS4"", ""FTS3"" ], ""extensions"": { ""json1"": null, ""spatialite"": ""5.0.1"" }, ""compile_options"": [ ""COMPILER=gcc-8.3.0"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_DBSTAT_VTAB"", ""ENABLE_FTS3"", ""ENABLE_FTS3_PARENTHESIS"", ""ENABLE_FTS3_TOKENIZER"", ""ENABLE_FTS4"", ""ENABLE_FTS5"", ""ENABLE_JSON1"", ""ENABLE_LOAD_EXTENSION"", ""ENABLE_PREUPDATE_HOOK"", ""ENABLE_RTREE"", ""ENABLE_SESSION"", ""ENABLE_STMTVTAB"", ""ENABLE_UNLOCK_NOTIFY"", ""ENABLE_UPDATE_DELETE_LIMIT"", ""HAVE_ISNAN"", ""LIKE_DOESNT_MATCH_BLOBS"", ""MAX_SCHEMA_RETRY=25"", ""MAX_VARIABLE_NUMBER=250000"", ""OMIT_LOOKASIDE"", ""SECURE_DELETE"", ""SOUNDEX"", ""TEMP_STORE=1"", ""THREADSAFE=1"", ""USE_URI"" ] } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069, https://github.com/simonw/datasette/issues/1272#issuecomment-808648974,https://api.github.com/repos/simonw/datasette/issues/1272,808648974,MDEyOklzc3VlQ29tbWVudDgwODY0ODk3NA==,9599,2021-03-27T04:29:42Z,2021-03-27T04:29:42Z,OWNER,"I'm skipping this for the moment because the new Dockerfile shape introduced in https://github.com/simonw/datasette/issues/1249#issuecomment-804404544 isn't compatible with this technique, since it installs Datasette from PyPI rather than directly from the repo. Will need to change that if I want to do this unit tests thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838245338, https://github.com/simonw/datasette/issues/1272#issuecomment-808647937,https://api.github.com/repos/simonw/datasette/issues/1272,808647937,MDEyOklzc3VlQ29tbWVudDgwODY0NzkzNw==,9599,2021-03-27T04:23:19Z,2021-03-27T04:23:36Z,OWNER,"Part of the challenge here is only running if a Docker daemon is available. I think this pattern works, in `tests/test_dockerfile.py`: ```python import httpx import pathlib import pytest import subprocess root = pathlib.Path(__file__).parent.parent def docker_is_available(): try: client = httpx.Client( transport=httpx.HTTPTransport(uds=""/var/run/docker.sock"") ) client.get(""http://docker/info"") return True except httpx.ConnectError: return False @pytest.fixture def build_container(): assert (root / ""Dockerfile"").exists() subprocess.check_call([ ""docker"", ""build"", str(root), ""-t"", ""datasette-dockerfile-test"" ]) @pytest.mark.skipif(not docker_is_available(), reason=""Docker is not available"" ) def test_dockerfile(build_container): output = subprocess.check_output([ ""docker"", ""run"", ""datasette-dockerfile-test"", ""datasette"", ""--get"", ""/_memory?sql=select+1&shape=_array"" ]) assert False, ""Implement better assertion here"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838245338, https://github.com/simonw/datasette/issues/1276#issuecomment-808642405,https://api.github.com/repos/simonw/datasette/issues/1276,808642405,MDEyOklzc3VlQ29tbWVudDgwODY0MjQwNQ==,9599,2021-03-27T03:53:18Z,2021-03-27T03:53:18Z,OWNER,That's really odd. What version of SQLite are you using on the server? You can tell by visiting `https://your-site/-/versions`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306, https://github.com/simonw/datasette/issues/1277#issuecomment-808641846,https://api.github.com/repos/simonw/datasette/issues/1277,808641846,MDEyOklzc3VlQ29tbWVudDgwODY0MTg0Ng==,9599,2021-03-27T03:49:34Z,2021-03-27T03:49:34Z,OWNER,"I fixed this already, it's a duplicate of #1239","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842212586, https://github.com/simonw/sqlite-utils/issues/252#issuecomment-808302971,https://api.github.com/repos/simonw/sqlite-utils/issues/252,808302971,MDEyOklzc3VlQ29tbWVudDgwODMwMjk3MQ==,9599,2021-03-26T15:21:38Z,2021-03-26T15:21:38Z,OWNER,Already got that! It's the `--nl` option - works for both importing and exporting data: https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842062949, https://github.com/simonw/sqlite-utils/issues/251#issuecomment-807647791,https://api.github.com/repos/simonw/sqlite-utils/issues/251,807647791,MDEyOklzc3VlQ29tbWVudDgwNzY0Nzc5MQ==,9599,2021-03-25T22:42:48Z,2021-03-25T22:44:31Z,OWNER,"Idea: enhance `lambda` to allow it to return a dictionary of values, which will then be used to populate new columns. Use a `--multicolumn` option to indicate this: sqlite-utils convert lambda mydb.db mytable mycolumn \ --code '{""first_name"": value.split()[0], ""last_name"": value.split()[1]}' \ --multicolumn --drop The `--drop` means ""drop the `mycolumn` column after making this change"". Maybe `--multi` is a better name than `--multicolumn` here, since either way it's going to need additional explanation somewhere. Would this overlap with #239 at all?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702, https://github.com/simonw/sqlite-utils/issues/251#issuecomment-807642041,https://api.github.com/repos/simonw/sqlite-utils/issues/251,807642041,MDEyOklzc3VlQ29tbWVudDgwNzY0MjA0MQ==,9599,2021-03-25T22:39:22Z,2021-03-25T22:39:22Z,OWNER,"Here's the full current implementation of that tool: https://github.com/simonw/sqlite-transform/blob/0.5/sqlite_transform/cli.py My current plan is to make this functionality available as the following: sqlite-utils convert jsonsplit mydb.db mytable mycolumn sqlite-utils convert parsedatetime mydb.db mytable mycolumn sqlite-utils convert parsedate mydb.db mytable mycolumn sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()' ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702, https://github.com/simonw/datasette/issues/741#issuecomment-806166575,https://api.github.com/repos/simonw/datasette/issues/741,806166575,MDEyOklzc3VlQ29tbWVudDgwNjE2NjU3NQ==,9599,2021-03-24T20:30:33Z,2021-03-24T20:30:33Z,OWNER,"`datasette package` is a mostly unmaintained feature at this point - it has a bit of test coverage but I've not made any improvements to it in a few years, and I don't use it for my own projects. I'll make this change to `package` at the same time as I land it for `publish` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136, https://github.com/simonw/datasette/issues/1274#issuecomment-805216038,https://api.github.com/repos/simonw/datasette/issues/1274,805216038,MDEyOklzc3VlQ29tbWVudDgwNTIxNjAzOA==,9599,2021-03-23T20:14:53Z,2021-03-23T20:14:53Z,OWNER,"Yes this is one of the main reasons I'm planning to switch to encouraging YAML be default instead of JSON (while still supporting JSON) - YAML supports comments and multi-line strings. See #1153 for YAML by default in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",839008371, https://github.com/simonw/datasette/issues/1153#issuecomment-805109341,https://api.github.com/repos/simonw/datasette/issues/1153,805109341,MDEyOklzc3VlQ29tbWVudDgwNTEwOTM0MQ==,9599,2021-03-23T17:55:48Z,2021-03-23T18:41:57Z,OWNER,"Beginnings of a UI element for switching between them: ```html
``` That `` has a padding of 12px, so using 12px padding on the tab links should get them to line up better.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454, https://github.com/simonw/datasette/issues/1153#issuecomment-805047117,https://api.github.com/repos/simonw/datasette/issues/1153,805047117,MDEyOklzc3VlQ29tbWVudDgwNTA0NzExNw==,9599,2021-03-23T16:30:15Z,2021-03-23T16:46:06Z,OWNER,"https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js is only 12.5KB zipped, 38KB total - so that's not a bad option. https://github.com/nodeca/js-yaml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454, https://github.com/simonw/datasette/issues/1270#issuecomment-805058241,https://api.github.com/repos/simonw/datasette/issues/1270,805058241,MDEyOklzc3VlQ29tbWVudDgwNTA1ODI0MQ==,9599,2021-03-23T16:45:39Z,2021-03-23T16:45:39Z,OWNER,"I managed to build SpatiaLite such that this isn't necessary any more. I'm still interested in pursuing this further though - it feels like it could be a more robust way of implementing timeouts, but I need to prove to myself that it's better (maybe better performance, or handles more edge-cases?). Not sure how to prove that yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092, https://github.com/simonw/datasette/issues/1153#issuecomment-805056806,https://api.github.com/repos/simonw/datasette/issues/1153,805056806,MDEyOklzc3VlQ29tbWVudDgwNTA1NjgwNg==,9599,2021-03-23T16:43:38Z,2021-03-23T16:43:38Z,OWNER,"I used this code to get that: ```javascript var jsonVersion = JSON.stringify(window.jsyaml.load(document.querySelector('.highlight-yaml').textContent), null, 4); div.querySelector('.highlight pre').innerText = jsonVersion; div.querySelector('.highlight pre').style.whiteSpace = 'pre-wrap' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454, https://github.com/simonw/datasette/issues/1153#issuecomment-805055291,https://api.github.com/repos/simonw/datasette/issues/1153,805055291,MDEyOklzc3VlQ29tbWVudDgwNTA1NTI5MQ==,9599,2021-03-23T16:41:31Z,2021-03-23T16:41:31Z,OWNER,"One downside of doing this conversion in JavaScript: it's much harder to get the same JSON syntax highlighting as that provided by Sphinx: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454, https://github.com/simonw/datasette/issues/1153#issuecomment-805050163,https://api.github.com/repos/simonw/datasette/issues/1153,805050163,MDEyOklzc3VlQ29tbWVudDgwNTA1MDE2Mw==,9599,2021-03-23T16:34:35Z,2021-03-23T16:35:32Z,OWNER,"https://docs.datasette.io/en/stable/metadata.html has this example: ```yaml title: Demonstrating Metadata from YAML description_html: |-This description includes a long HTML string
This demonstrates basic LIKE search ``` I ran this in the browser dev tools: ```javascript var s = document.createElement('script') s.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js' document.head.appendChild(s) var yamlExample = document.querySelector('.highlight-yaml').textContent); console.log(JSON.stringify(window.jsyaml.load(yamlExample), null, 4)) ``` And got: ```json { ""title"": ""Demonstrating Metadata from YAML"", ""description_html"": ""
This description includes a long HTML string
\nThis demonstrates basic LIKE search""
}
}
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/issues/1153#issuecomment-805042880,https://api.github.com/repos/simonw/datasette/issues/1153,805042880,MDEyOklzc3VlQ29tbWVudDgwNTA0Mjg4MA==,9599,2021-03-23T16:24:32Z,2021-03-23T16:24:32Z,OWNER,... actually I think I would do that conversion in Python. The client-side YAML parsers all look a little bit heavy to me in terms of additional page weight.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/issues/1153#issuecomment-805041522,https://api.github.com/repos/simonw/datasette/issues/1153,805041522,MDEyOklzc3VlQ29tbWVudDgwNTA0MTUyMg==,9599,2021-03-23T16:22:46Z,2021-03-23T16:22:46Z,OWNER,"That's a good idea. I could do that with JavaScript - loading YAML and converting it to JSON in JavaScript shouldn't be hard, and it's better than JSON-to-YAML because there's only one correct JSON representation of a YAML file whereas you can represent a JSON document in YAML in a bunch of different ways.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/issues/1249#issuecomment-805033155,https://api.github.com/repos/simonw/datasette/issues/1249,805033155,MDEyOklzc3VlQ29tbWVudDgwNTAzMzE1NQ==,9599,2021-03-23T16:12:13Z,2021-03-23T16:12:13Z,OWNER,"Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source
> The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful [VirtualKNN module](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN)), take a look at the [Datasette Dockerfile](https://github.com/simonw/datasette/blob/master/Dockerfile).
See also #1273","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/163#issuecomment-804540869,https://api.github.com/repos/simonw/datasette/issues/163,804540869,MDEyOklzc3VlQ29tbWVudDgwNDU0MDg2OQ==,9599,2021-03-23T02:44:33Z,2021-03-23T02:44:33Z,OWNER,Comments welcome!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",279547886,
https://github.com/simonw/datasette/pull/1271#issuecomment-804299406,https://api.github.com/repos/simonw/datasette/issues/1271,804299406,MDEyOklzc3VlQ29tbWVudDgwNDI5OTQwNg==,9599,2021-03-22T18:36:14Z,2021-03-22T21:49:27Z,OWNER,"This isn't actually working - the outer code attempts to send an `.interrupt()` call to the connection object via the `connections` thread-local, which doesn't work because it's a thread-local so the connection isn't visible to that code.
Need to figure out how to communicate with that thread properly.
Also a test that fails in this particular case would be a good idea!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,
https://github.com/simonw/datasette/issues/1249#issuecomment-804406675,https://api.github.com/repos/simonw/datasette/issues/1249,804406675,MDEyOklzc3VlQ29tbWVudDgwNDQwNjY3NQ==,9599,2021-03-22T21:26:27Z,2021-03-22T21:26:27Z,OWNER,(Without the `apt-get update ...` SpatiaLite line it's 125MB),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804404544,https://api.github.com/repos/simonw/datasette/issues/1249,804404544,MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA==,9599,2021-03-22T21:22:56Z,2021-03-22T21:24:24Z,OWNER,"Final version of Dockerfile which installs the specified version from GitHub:
docker build . -t datasette-spatialite --build-arg VERSION=0.55
```dockerfile
FROM python:3.9.2-slim-buster as build
# Version of Datasette to install, e.g. 0.55
# docker build . -t datasette --build-arg VERSION=0.55
ARG VERSION
# software-properties-common provides add-apt-repository
# which we need in order to install a more recent release
# of libsqlite3-mod-spatialite from the sid distribution
RUN apt-get update && \
apt-get -y --no-install-recommends install software-properties-common && \
add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
apt-get update && \
apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
apt-get remove -y software-properties-common && \
apt clean && \
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg
RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \
find /usr/local/lib -name '__pycache__' | xargs rm -r && \
rm -rf /root/.cache/pip
EXPOSE 8001
CMD [""datasette""]
```
Run against 0.55 this produces an image of 262MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804384196,https://api.github.com/repos/simonw/datasette/issues/1249,804384196,MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng==,9599,2021-03-22T20:48:46Z,2021-03-22T20:48:46Z,OWNER,I think part of the reason it's smaller is that I ran `pip install datasette` instead of using `COPY . /datasette` followed by `pip install /datasette`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804380181,https://api.github.com/repos/simonw/datasette/issues/1249,804380181,MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ==,9599,2021-03-22T20:42:16Z,2021-03-22T20:42:16Z,OWNER,"Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804379644,https://api.github.com/repos/simonw/datasette/issues/1249,804379644,MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA==,9599,2021-03-22T20:41:23Z,2021-03-22T20:41:23Z,OWNER,I tried adding `apt-get remove -y software-properties-common &&` to remove `software-properties-common` but it made no difference to the image size.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804372977,https://api.github.com/repos/simonw/datasette/issues/1249,804372977,MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw==,9599,2021-03-22T20:30:37Z,2021-03-22T20:30:37Z,OWNER,"I tried copying just the `mod_spatialite.so` file into a second stage build but it failed. So I ran `bash` in a working image and used `ldd` to figure out what it was linked to:
```
root@39683f91e588:/usr/lib/x86_64-linux-gnu# ldd mod_spatialite.so
linux-vdso.so.1 (0x00007ffd021f4000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f5c75412000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c753f0000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c752ac000)
libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f5c750a0000)
librttopo.so.1 => /usr/lib/x86_64-linux-gnu/librttopo.so.1 (0x00007f5c75028000)
libfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f5c7501c000)
libproj.so.19 => /usr/lib/x86_64-linux-gnu/libproj.so.19 (0x00007f5c74ca7000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c74a89000)
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f5c74967000)
libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f5c7492b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c74766000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c74760000)
libicuuc.so.67 => /usr/lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007f5c74575000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5c7454d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5c75d49000)
libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f5c744c7000)
libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f5c74439000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5c7426c000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c74250000)
libgeos-3.9.0.so => /usr/lib/x86_64-linux-gnu/libgeos-3.9.0.so (0x00007f5c74040000)
libicudata.so.67 => /usr/lib/x86_64-linux-gnu/libicudata.so.67 (0x00007f5c72527000)
libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f5c724bc000)
libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f5c7241c000)
libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f5c7220e000)
libjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f5c72188000)
libdeflate.so.0 => /usr/lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f5c7216c000)
libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f5c72144000)
libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f5c72125000)
librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f5c71f08000)
libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f5c71eda000)
libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f5c71ec5000)
libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f5c71e8d000)
libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f5c71ce0000)
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f5c71c93000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f5c71bb3000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f5c71b7f000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f5c71b77000)
libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f5c71b23000)
liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f5c71b12000)
libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f5c7198e000)
libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f5c71955000)
libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f5c718d0000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f5c717b2000)
libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f5c71683000)
libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f5c71470000)
libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f5c71461000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f5c71458000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5c7143e000)
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f5c71421000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5c713fe000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f5c713f4000)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804368372,https://api.github.com/repos/simonw/datasette/issues/1249,804368372,MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg==,9599,2021-03-22T20:22:43Z,2021-03-22T20:22:43Z,OWNER,"```dockerfile
FROM python:3.9.2-slim-buster as build
# software-properties-common provides add-apt-repository
RUN apt-get update && \
apt-get -y --no-install-recommends install software-properties-common && \
add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
apt-get update && \
apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg
RUN pip install datasette && \
find /usr/local/lib -name '__pycache__' | xargs rm -r && \
rm -rf /root/.cache/pip
EXPOSE 8001
CMD [""datasette""]
```
262 MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804363687,https://api.github.com/repos/simonw/datasette/issues/1249,804363687,MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw==,9599,2021-03-22T20:15:00Z,2021-03-22T20:15:00Z,OWNER,"```
RUN pip install datasette && \
find /usr/local/lib -name '__pycache__' | xargs rm -r
```
That dropped it to 265MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804360701,https://api.github.com/repos/simonw/datasette/issues/1249,804360701,MDEyOklzc3VlQ29tbWVudDgwNDM2MDcwMQ==,9599,2021-03-22T20:10:07Z,2021-03-22T20:10:07Z,OWNER,Adding `--no-install-recommends` dropped it to 275MB,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804347152,https://api.github.com/repos/simonw/datasette/issues/1249,804347152,MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg==,9599,2021-03-22T19:47:56Z,2021-03-22T19:48:03Z,OWNER,I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804344553,https://api.github.com/repos/simonw/datasette/issues/1249,804344553,MDEyOklzc3VlQ29tbWVudDgwNDM0NDU1Mw==,9599,2021-03-22T19:43:25Z,2021-03-22T19:43:25Z,OWNER,Does `--no-install-recommends` make a difference?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804338678,https://api.github.com/repos/simonw/datasette/issues/1249,804338678,MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA==,9599,2021-03-22T19:33:43Z,2021-03-22T19:33:43Z,OWNER,"Replacing `rm -rf /var/lib/{apt,dpkg,cache,log}/` with
```
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg
```
Got the size down to 305MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804318314,https://api.github.com/repos/simonw/datasette/issues/1249,804318314,MDEyOklzc3VlQ29tbWVudDgwNDMxODMxNA==,9599,2021-03-22T19:04:30Z,2021-03-22T19:04:30Z,OWNER,Considering the image on Docker Hub right now is `383MB` this is actually an improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804317545,https://api.github.com/repos/simonw/datasette/issues/1249,804317545,MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ==,9599,2021-03-22T19:03:22Z,2021-03-22T19:03:22Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build
# software-properties-common provides add-apt-repository
RUN apt-get update && \
apt-get -y install software-properties-common && \
add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
apt-get update && \
apt-get -t sid install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
RUN pip install datasette
EXPOSE 8001
CMD [""datasette""]
```
Produces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804310353,https://api.github.com/repos/simonw/datasette/issues/1249,804310353,MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw==,9599,2021-03-22T18:52:12Z,2021-03-22T18:52:12Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build
# Setup build dependencies
RUN apt update \
&& apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
&& apt clean
RUN wget ""https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz"" && tar xzf sqlite-autoconf-3340100.tar.gz \
&& cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
CFLAGS=""-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1"" \
&& make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz"" && tar zxf freexl-1.0.6.tar.gz \
&& cd freexl-1.0.6 && ./configure && make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz"" && tar zxf libspatialite-5.0.1.tar.gz \
&& cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz"" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz"" && tar zxf spatialite-tools-5.0.0.tar.gz \
&& cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install
# Add local code to the image instead of fetching from pypi.
#COPY . /datasette
#RUN pip install /datasette
RUN pip install datasette
FROM python:3.9.2-slim-buster
# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```
Produced a 448MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804309510,https://api.github.com/repos/simonw/datasette/issues/1249,804309510,MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA==,9599,2021-03-22T18:50:50Z,2021-03-22T18:50:50Z,OWNER,"Ideally I'd like to use the Debian stable `python:3.9.2-slim-buster` base image but install SpatiaLite from Debian unstable here: https://packages.debian.org/sid/libspatialite7
This pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/pull/1271#issuecomment-804265042,https://api.github.com/repos/simonw/datasette/issues/1271,804265042,MDEyOklzc3VlQ29tbWVudDgwNDI2NTA0Mg==,9599,2021-03-22T17:45:45Z,2021-03-22T17:45:45Z,OWNER,"I can remove this code too:
https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L190-L192","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,
https://github.com/simonw/datasette/issues/1249#issuecomment-804263434,https://api.github.com/repos/simonw/datasette/issues/1249,804263434,MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA==,9599,2021-03-22T17:43:25Z,2021-03-22T17:43:25Z,OWNER,I figured out the cause of the hang in #1268 - it was caused by `select count(*) from SpatialIndex` interacting badly with the `set_progress_handler()` mechanism I was using to implement query time limits. #1271 has a replacement for that using `asyncio.wait_for()` and `conn.interrupt()` which should resolve the SpatiaLite issue too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1268#issuecomment-804261915,https://api.github.com/repos/simonw/datasette/issues/1268,804261915,MDEyOklzc3VlQ29tbWVudDgwNDI2MTkxNQ==,9599,2021-03-22T17:41:12Z,2021-03-22T17:41:12Z,OWNER,"Closing this because I've figured out the root of the problem now, and I have a potential solution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1269#issuecomment-804261610,https://api.github.com/repos/simonw/datasette/issues/1269,804261610,MDEyOklzc3VlQ29tbWVudDgwNDI2MTYxMA==,9599,2021-03-22T17:40:41Z,2021-03-22T17:40:41Z,OWNER,"#1270 looks promising, and I don't want to leave open a security hole where someone could potentially hang Datasette with a nasty `count(*)` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,
https://github.com/simonw/datasette/issues/1270#issuecomment-804255633,https://api.github.com/repos/simonw/datasette/issues/1270,804255633,MDEyOklzc3VlQ29tbWVudDgwNDI1NTYzMw==,9599,2021-03-22T17:32:02Z,2021-03-22T17:32:08Z,OWNER,Confirmed that the `interrupt()` based cancellation mechanism fixes the SpatiaLite issue in #1268!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,
https://github.com/simonw/datasette/issues/1270#issuecomment-803834784,https://api.github.com/repos/simonw/datasette/issues/1270,803834784,MDEyOklzc3VlQ29tbWVudDgwMzgzNDc4NA==,9599,2021-03-22T07:31:57Z,2021-03-22T16:22:19Z,OWNER,"I think the implementation for this goes here: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L146-L157
I figured out a similar pattern in `datasette-ripgrep` here: https://github.com/simonw/datasette-ripgrep/blob/0.7/datasette_ripgrep/__init__.py#L63-L71","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,
https://github.com/simonw/datasette/issues/1268#issuecomment-803802957,https://api.github.com/repos/simonw/datasette/issues/1268,803802957,MDEyOklzc3VlQ29tbWVudDgwMzgwMjk1Nw==,9599,2021-03-22T06:38:14Z,2021-03-22T06:38:14Z,OWNER,"Also worth trying is to change this code:
```python
n = 1000
if ms < 50:
n = 1
```
What happens with `n = 10` instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1269#issuecomment-803785808,https://api.github.com/repos/simonw/datasette/issues/1269,803785808,MDEyOklzc3VlQ29tbWVudDgwMzc4NTgwOA==,9599,2021-03-22T06:00:53Z,2021-03-22T06:00:53Z,OWNER,This may not be necessary if using `.interrupt() for SQLite timeouts in #1270 works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,
https://github.com/simonw/datasette/issues/1268#issuecomment-803784902,https://api.github.com/repos/simonw/datasette/issues/1268,803784902,MDEyOklzc3VlQ29tbWVudDgwMzc4NDkwMg==,9599,2021-03-22T05:59:06Z,2021-03-22T05:59:06Z,OWNER,"Even if I implement that workaround in #1269 I'm concerned that this could still allow users to deliberately crash Datasette (if it's running SpatiaLite 5.0) by executing `select count(*) from SpatialIndex`.
That `interrupt` timeout mechanism is worth digging into further.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803782705,https://api.github.com/repos/simonw/datasette/issues/1268,803782705,MDEyOklzc3VlQ29tbWVudDgwMzc4MjcwNQ==,9599,2021-03-22T05:54:19Z,2021-03-22T05:54:19Z,OWNER,"Got two new TILs out of this:
* [Tracing every executed Python statement](https://til.simonwillison.net/python/tracing-every-statement)
* [Running gdb against a Python process in a running Docker container](https://til.simonwillison.net/docker/gdb-python-docker)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803777724,https://api.github.com/repos/simonw/datasette/issues/1268,803777724,MDEyOklzc3VlQ29tbWVudDgwMzc3NzcyNA==,9599,2021-03-22T05:42:50Z,2021-03-22T05:43:23Z,OWNER,"
If I want to avoid counting virtual tables, I need to detect which tables are virtual tables.
The safest way to do this is probably to pull the `sql` for every table and then, in Python, check for values that start with `create virtual table` after converting to lower case, using any number of spaces.
This would catch things like ` CREATE virtual TABLE` which might be missed by a SQL `like` query. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803775121,https://api.github.com/repos/simonw/datasette/issues/1268,803775121,MDEyOklzc3VlQ29tbWVudDgwMzc3NTEyMQ==,9599,2021-03-22T05:36:26Z,2021-03-22T05:36:26Z,OWNER,So one fix could be to avoid running counts for anything that turns out to be a virtual table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774926,https://api.github.com/repos/simonw/datasette/issues/1268,803774926,MDEyOklzc3VlQ29tbWVudDgwMzc3NDkyNg==,9599,2021-03-22T05:35:56Z,2021-03-22T05:35:56Z,OWNER,That's in this code here: https://github.com/simonw/datasette/blob/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b/datasette/database.py#L221-L241,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774518,https://api.github.com/repos/simonw/datasette/issues/1268,803774518,MDEyOklzc3VlQ29tbWVudDgwMzc3NDUxOA==,9599,2021-03-22T05:34:57Z,2021-03-22T05:34:57Z,OWNER,"... and sure enough, adding this code fixed the problem:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 3579cce..b466b12 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -224,6 +226,9 @@ class Database:
# Try to get counts for each table, $limit timeout for each count
counts = {}
for table in await self.table_names():
+ if table == ""SpatialIndex"":
+ counts[table] = 0
+ continue
try:
table_count = (
await self.execute(
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803773484,https://api.github.com/repos/simonw/datasette/issues/1268,803773484,MDEyOklzc3VlQ29tbWVudDgwMzc3MzQ4NA==,9599,2021-03-22T05:32:29Z,2021-03-22T05:32:29Z,OWNER,"To figure out which SQL query triggers the problem I added this code to write to a log file:
```python
with sqlite_timelimit(conn, time_limit_ms):
try:
cursor = conn.cursor()
with open(""/tmp/sql.log"", ""ab"", buffering=0) as fp:
fp.write((""{}: {}\n"".format(sql, params)).encode(""utf-8""))
cursor.execute(sql, params if params is not None else {})
```
I had to use `ab` binary mode because Python doesn't allow `buffering=0` for non-binary file operations.
With the log enabled, I used `docker exec -it 589ae68de943 bash` to attach to the running container and `tail -f /tmp/sql.log` to see the logs. Here's where it broke:
```
select count(*) from [idx_civici_geom_parent]: None
select count(*) from [sqlite_stat1]: None
select count(*) from [sqlite_stat3]: None
select count(*) from [SpatialIndex]: None
```
So attempting to run a `count(*)` against the `SpatialIndex` virtual table is the thing that triggers the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764919,https://api.github.com/repos/simonw/datasette/issues/1268,803764919,MDEyOklzc3VlQ29tbWVudDgwMzc2NDkxOQ==,9599,2021-03-22T05:11:11Z,2021-03-22T05:11:11Z,OWNER,"Maybe I could implement SQLite query timeouts using the `interrupt()` method instead of the progress handler hack I'm currently using?
https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764200,https://api.github.com/repos/simonw/datasette/issues/1268,803764200,MDEyOklzc3VlQ29tbWVudDgwMzc2NDIwMA==,9599,2021-03-22T05:09:13Z,2021-03-22T05:09:13Z,OWNER,"I tried building a container where the `conn.set_progress_handler(handler, n)` line was commented out... and it fixed the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762969,https://api.github.com/repos/simonw/datasette/issues/1268,803762969,MDEyOklzc3VlQ29tbWVudDgwMzc2Mjk2OQ==,9599,2021-03-22T05:05:51Z,2021-03-22T05:05:51Z,OWNER,I had to run `docker kill 16197781a7b5` to kill the broken container - Ctrl+C in the Datasette console window didn't do anything.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762609,https://api.github.com/repos/simonw/datasette/issues/1268,803762609,MDEyOklzc3VlQ29tbWVudDgwMzc2MjYwOQ==,9599,2021-03-22T05:05:00Z,2021-03-22T05:05:00Z,OWNER,"Using https://til.simonwillison.net/docker/attach-bash-to-running-container - I figured out how to run `gdb`. I had to use `--privileged` here because otherwise `gdb` showed a ""Could not attach to process"" error.
```
docker exec --privileged -it 16197781a7b5 bash
# apt-get install gdb python3-dbg
# gdb /usr/bin/python3 -p 20
```
This paused the process. I tried running this:
```
(gdb) py-bt
Traceback (most recent call first):
File ""/usr/lib/python3.8/asyncio/base_events.py"", line 1845, in _run_once
if handle._cancelled:
File ""/usr/lib/python3.8/asyncio/base_events.py"", line 570, in run_forever
self._run_once()
File ""/usr/lib/python3.8/asyncio/base_events.py"", line 603, in run_until_complete
self.run_forever()
File ""/usr/local/lib/python3.8/dist-packages/uvicorn/server.py"", line 49, in run
loop.run_until_complete(self.serve(sockets=sockets))
File ""/usr/local/lib/python3.8/dist-packages/uvicorn/main.py"", line 386, in run
server.run()
File ""/usr/local/lib/python3.8/dist-packages/datasette/cli.py"", line 575, in serve
uvicorn.run(ds.app(), **uvicorn_kwargs)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/usr/local/bin/datasette"", line 8, in