.html file that hardcodes a default param in the link, but I'd rather not customize the template just for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,
https://github.com/simonw/datasette/issues/741#issuecomment-806166575,https://api.github.com/repos/simonw/datasette/issues/741,806166575,MDEyOklzc3VlQ29tbWVudDgwNjE2NjU3NQ==,9599,2021-03-24T20:30:33Z,2021-03-24T20:30:33Z,OWNER,"`datasette package` is a mostly unmaintained feature at this point - it has a bit of test coverage but I've not made any improvements to it in a few years, and I don't use it for my own projects.
I'll make this change to `package` at the same time as I land it for `publish` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,
https://github.com/simonw/datasette/issues/741#issuecomment-806010960,https://api.github.com/repos/simonw/datasette/issues/741,806010960,MDEyOklzc3VlQ29tbWVudDgwNjAxMDk2MA==,596279,2021-03-24T17:19:42Z,2021-03-24T17:19:42Z,NONE,"Ah, okay so `--extra-options` applies to both `datasette publish` and `datasette package`? There wren't any examples of it being used with `publish` in the docs, so this tripped me up for a bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,
https://github.com/simonw/datasette/issues/1274#issuecomment-805216038,https://api.github.com/repos/simonw/datasette/issues/1274,805216038,MDEyOklzc3VlQ29tbWVudDgwNTIxNjAzOA==,9599,2021-03-23T20:14:53Z,2021-03-23T20:14:53Z,OWNER,"Yes this is one of the main reasons I'm planning to switch to encouraging YAML be default instead of JSON (while still supporting JSON) - YAML supports comments and multi-line strings.
See #1153 for YAML by default in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",839008371,
https://github.com/simonw/datasette/issues/1274#issuecomment-805214307,https://api.github.com/repos/simonw/datasette/issues/1274,805214307,MDEyOklzc3VlQ29tbWVudDgwNTIxNDMwNw==,7476523,2021-03-23T20:12:29Z,2021-03-23T20:12:29Z,CONTRIBUTOR,"One issue I could see with adding first class support for metadata in hjson format is that this would require adding an additional dependency to handle this, for a feature that would be unused by many users. I wonder if this could fit in as a plugin instead; if a hook existed for loading metadata (maybe as part of https://github.com/simonw/datasette/issues/860) the metadata could then come from any source, as specified by plugins, e.g. hjson, toml, XML, a database table etc.
Until/unless this exists, a few ideas for how you could add comments:
- Using YAML as you suggest.
- A common pattern is adding a `""comment""` key for comments to any object in JSON - I don't think including an unnecessary key like this would break anything in Datasette, but not certain.
- You could use another tool as a preprocessor for your JSON metadata - e.g. hjson or Jsonnet. You'd write the metadata in that format, and then convert that into JSON to actually use as your final metadata.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",839008371,
https://github.com/simonw/datasette/issues/1153#issuecomment-805109341,https://api.github.com/repos/simonw/datasette/issues/1153,805109341,MDEyOklzc3VlQ29tbWVudDgwNTEwOTM0MQ==,9599,2021-03-23T17:55:48Z,2021-03-23T18:41:57Z,OWNER,"Beginnings of a UI element for switching between them:
```html
```
![]()
That `` has a padding of 12px, so using 12px padding on the tab links should get them to line up better.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/issues/1153#issuecomment-805047117,https://api.github.com/repos/simonw/datasette/issues/1153,805047117,MDEyOklzc3VlQ29tbWVudDgwNTA0NzExNw==,9599,2021-03-23T16:30:15Z,2021-03-23T16:46:06Z,OWNER,"https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js is only 12.5KB zipped, 38KB total - so that's not a bad option.
https://github.com/nodeca/js-yaml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/issues/1270#issuecomment-805058241,https://api.github.com/repos/simonw/datasette/issues/1270,805058241,MDEyOklzc3VlQ29tbWVudDgwNTA1ODI0MQ==,9599,2021-03-23T16:45:39Z,2021-03-23T16:45:39Z,OWNER,"I managed to build SpatiaLite such that this isn't necessary any more. I'm still interested in pursuing this further though - it feels like it could be a more robust way of implementing timeouts, but I need to prove to myself that it's better (maybe better performance, or handles more edge-cases?). Not sure how to prove that yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,
https://github.com/simonw/datasette/issues/1153#issuecomment-805056806,https://api.github.com/repos/simonw/datasette/issues/1153,805056806,MDEyOklzc3VlQ29tbWVudDgwNTA1NjgwNg==,9599,2021-03-23T16:43:38Z,2021-03-23T16:43:38Z,OWNER,"I used this code to get that:
```javascript
var jsonVersion = JSON.stringify(window.jsyaml.load(document.querySelector('.highlight-yaml').textContent), null, 4);
div.querySelector('.highlight pre').innerText = jsonVersion;
div.querySelector('.highlight pre').style.whiteSpace = 'pre-wrap'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/issues/1153#issuecomment-805055291,https://api.github.com/repos/simonw/datasette/issues/1153,805055291,MDEyOklzc3VlQ29tbWVudDgwNTA1NTI5MQ==,9599,2021-03-23T16:41:31Z,2021-03-23T16:41:31Z,OWNER,"One downside of doing this conversion in JavaScript: it's much harder to get the same JSON syntax highlighting as that provided by Sphinx:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/issues/1153#issuecomment-805050163,https://api.github.com/repos/simonw/datasette/issues/1153,805050163,MDEyOklzc3VlQ29tbWVudDgwNTA1MDE2Mw==,9599,2021-03-23T16:34:35Z,2021-03-23T16:35:32Z,OWNER,"https://docs.datasette.io/en/stable/metadata.html has this example:
```yaml
title: Demonstrating Metadata from YAML
description_html: |-
This description includes a long HTML string
- YAML is better for embedding HTML strings than JSON!
license: ODbL
license_url: https://opendatacommons.org/licenses/odbl/
databases:
fixtures:
tables:
no_primary_key:
hidden: true
queries:
neighborhood_search:
sql: |-
select neighborhood, facet_cities.name, state
from facetable join facet_cities on facetable.city_id = facet_cities.id
where neighborhood like '%' || :text || '%' order by neighborhood;
title: Search neighborhoods
description_html: |-
This demonstrates basic LIKE search
```
I ran this in the browser dev tools:
```javascript
var s = document.createElement('script')
s.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js'
document.head.appendChild(s)
var yamlExample = document.querySelector('.highlight-yaml').textContent);
console.log(JSON.stringify(window.jsyaml.load(yamlExample), null, 4))
```
And got:
```json
{
""title"": ""Demonstrating Metadata from YAML"",
""description_html"": ""
This description includes a long HTML string
\n\n - YAML is better for embedding HTML strings than JSON!
\n
"",
""license"": ""ODbL"",
""license_url"": ""https://opendatacommons.org/licenses/odbl/"",
""databases"": {
""fixtures"": {
""tables"": {
""no_primary_key"": {
""hidden"": true
}
},
""queries"": {
""neighborhood_search"": {
""sql"": ""select neighborhood, facet_cities.name, state\nfrom facetable join facet_cities on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%' order by neighborhood;"",
""title"": ""Search neighborhoods"",
""description_html"": ""This demonstrates basic LIKE search""
}
}
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/issues/1153#issuecomment-805042880,https://api.github.com/repos/simonw/datasette/issues/1153,805042880,MDEyOklzc3VlQ29tbWVudDgwNTA0Mjg4MA==,9599,2021-03-23T16:24:32Z,2021-03-23T16:24:32Z,OWNER,... actually I think I would do that conversion in Python. The client-side YAML parsers all look a little bit heavy to me in terms of additional page weight.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/issues/1153#issuecomment-805041522,https://api.github.com/repos/simonw/datasette/issues/1153,805041522,MDEyOklzc3VlQ29tbWVudDgwNTA0MTUyMg==,9599,2021-03-23T16:22:46Z,2021-03-23T16:22:46Z,OWNER,"That's a good idea. I could do that with JavaScript - loading YAML and converting it to JSON in JavaScript shouldn't be hard, and it's better than JSON-to-YAML because there's only one correct JSON representation of a YAML file whereas you can represent a JSON document in YAML in a bunch of different ways.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/issues/1249#issuecomment-805033155,https://api.github.com/repos/simonw/datasette/issues/1249,805033155,MDEyOklzc3VlQ29tbWVudDgwNTAzMzE1NQ==,9599,2021-03-23T16:12:13Z,2021-03-23T16:12:13Z,OWNER,"Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source
> The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful [VirtualKNN module](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN)), take a look at the [Datasette Dockerfile](https://github.com/simonw/datasette/blob/master/Dockerfile).
See also #1273","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/pull/1159#issuecomment-804698315,https://api.github.com/repos/simonw/datasette/issues/1159,804698315,MDEyOklzc3VlQ29tbWVudDgwNDY5ODMxNQ==,552629,2021-03-23T07:58:28Z,2021-03-23T07:58:38Z,NONE,@mroswell Did you try it with more columns ? The display is flexible and columns get closer as new ones are added.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",774332247,
https://github.com/simonw/datasette/issues/1153#issuecomment-804640440,https://api.github.com/repos/simonw/datasette/issues/1153,804640440,MDEyOklzc3VlQ29tbWVudDgwNDY0MDQ0MA==,192568,2021-03-23T05:58:20Z,2021-03-23T05:58:20Z,CONTRIBUTOR,Could there be a little widget that offers conversion from one to the other? ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,
https://github.com/simonw/datasette/pull/1159#issuecomment-804639427,https://api.github.com/repos/simonw/datasette/issues/1159,804639427,MDEyOklzc3VlQ29tbWVudDgwNDYzOTQyNw==,192568,2021-03-23T05:56:02Z,2021-03-23T05:56:02Z,CONTRIBUTOR,"With just three facets, I like it, but it does take more horizontal space. Would be nice to have a switch somewhere, enabling either original compact option or this proposed more-readable option. Also some control over word wrap (width setting) and facet spacing. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",774332247,
https://github.com/simonw/datasette/issues/164#issuecomment-804541064,https://api.github.com/repos/simonw/datasette/issues/164,804541064,MDEyOklzc3VlQ29tbWVudDgwNDU0MTA2NA==,192568,2021-03-23T02:45:12Z,2021-03-23T02:45:12Z,CONTRIBUTOR,"""datasette skeleton"" feature removed #476","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",280013907,
https://github.com/simonw/datasette/issues/163#issuecomment-804540869,https://api.github.com/repos/simonw/datasette/issues/163,804540869,MDEyOklzc3VlQ29tbWVudDgwNDU0MDg2OQ==,9599,2021-03-23T02:44:33Z,2021-03-23T02:44:33Z,OWNER,Comments welcome!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",279547886,
https://github.com/simonw/datasette/issues/163#issuecomment-804539729,https://api.github.com/repos/simonw/datasette/issues/163,804539729,MDEyOklzc3VlQ29tbWVudDgwNDUzOTcyOQ==,192568,2021-03-23T02:41:14Z,2021-03-23T02:41:14Z,CONTRIBUTOR,"I'm visiting old issues for context while learning datasette. Let me know if okay to make the occasional comment like this one.
querystring argument now located at:
https://docs.datasette.io/en/latest/settings.html#sql-time-limit-ms","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",279547886,
https://github.com/simonw/datasette/issues/88#issuecomment-804471733,https://api.github.com/repos/simonw/datasette/issues/88,804471733,MDEyOklzc3VlQ29tbWVudDgwNDQ3MTczMw==,192568,2021-03-22T23:46:36Z,2021-03-22T23:46:36Z,CONTRIBUTOR,Google Map API limits seem to prevent https://nhs-england-map.netlify.com from being a working demo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273775212,
https://github.com/simonw/datasette/pull/1271#issuecomment-804299406,https://api.github.com/repos/simonw/datasette/issues/1271,804299406,MDEyOklzc3VlQ29tbWVudDgwNDI5OTQwNg==,9599,2021-03-22T18:36:14Z,2021-03-22T21:49:27Z,OWNER,"This isn't actually working - the outer code attempts to send an `.interrupt()` call to the connection object via the `connections` thread-local, which doesn't work because it's a thread-local so the connection isn't visible to that code.
Need to figure out how to communicate with that thread properly.
Also a test that fails in this particular case would be a good idea!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,
https://github.com/simonw/datasette/issues/1149#issuecomment-804415619,https://api.github.com/repos/simonw/datasette/issues/1149,804415619,MDEyOklzc3VlQ29tbWVudDgwNDQxNTYxOQ==,192568,2021-03-22T21:43:16Z,2021-03-22T21:43:16Z,CONTRIBUTOR,Sounds like a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769520939,
https://github.com/simonw/datasette/issues/1249#issuecomment-804406675,https://api.github.com/repos/simonw/datasette/issues/1249,804406675,MDEyOklzc3VlQ29tbWVudDgwNDQwNjY3NQ==,9599,2021-03-22T21:26:27Z,2021-03-22T21:26:27Z,OWNER,(Without the `apt-get update ...` SpatiaLite line it's 125MB),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804404544,https://api.github.com/repos/simonw/datasette/issues/1249,804404544,MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA==,9599,2021-03-22T21:22:56Z,2021-03-22T21:24:24Z,OWNER,"Final version of Dockerfile which installs the specified version from GitHub:
docker build . -t datasette-spatialite --build-arg VERSION=0.55
```dockerfile
FROM python:3.9.2-slim-buster as build
# Version of Datasette to install, e.g. 0.55
# docker build . -t datasette --build-arg VERSION=0.55
ARG VERSION
# software-properties-common provides add-apt-repository
# which we need in order to install a more recent release
# of libsqlite3-mod-spatialite from the sid distribution
RUN apt-get update && \
apt-get -y --no-install-recommends install software-properties-common && \
add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
apt-get update && \
apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
apt-get remove -y software-properties-common && \
apt clean && \
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg
RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \
find /usr/local/lib -name '__pycache__' | xargs rm -r && \
rm -rf /root/.cache/pip
EXPOSE 8001
CMD [""datasette""]
```
Run against 0.55 this produces an image of 262MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804384196,https://api.github.com/repos/simonw/datasette/issues/1249,804384196,MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng==,9599,2021-03-22T20:48:46Z,2021-03-22T20:48:46Z,OWNER,I think part of the reason it's smaller is that I ran `pip install datasette` instead of using `COPY . /datasette` followed by `pip install /datasette`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804380181,https://api.github.com/repos/simonw/datasette/issues/1249,804380181,MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ==,9599,2021-03-22T20:42:16Z,2021-03-22T20:42:16Z,OWNER,"Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804379644,https://api.github.com/repos/simonw/datasette/issues/1249,804379644,MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA==,9599,2021-03-22T20:41:23Z,2021-03-22T20:41:23Z,OWNER,I tried adding `apt-get remove -y software-properties-common &&` to remove `software-properties-common` but it made no difference to the image size.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804372977,https://api.github.com/repos/simonw/datasette/issues/1249,804372977,MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw==,9599,2021-03-22T20:30:37Z,2021-03-22T20:30:37Z,OWNER,"I tried copying just the `mod_spatialite.so` file into a second stage build but it failed. So I ran `bash` in a working image and used `ldd` to figure out what it was linked to:
```
root@39683f91e588:/usr/lib/x86_64-linux-gnu# ldd mod_spatialite.so
linux-vdso.so.1 (0x00007ffd021f4000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f5c75412000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c753f0000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c752ac000)
libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f5c750a0000)
librttopo.so.1 => /usr/lib/x86_64-linux-gnu/librttopo.so.1 (0x00007f5c75028000)
libfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f5c7501c000)
libproj.so.19 => /usr/lib/x86_64-linux-gnu/libproj.so.19 (0x00007f5c74ca7000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c74a89000)
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f5c74967000)
libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f5c7492b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c74766000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c74760000)
libicuuc.so.67 => /usr/lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007f5c74575000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5c7454d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5c75d49000)
libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f5c744c7000)
libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f5c74439000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5c7426c000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c74250000)
libgeos-3.9.0.so => /usr/lib/x86_64-linux-gnu/libgeos-3.9.0.so (0x00007f5c74040000)
libicudata.so.67 => /usr/lib/x86_64-linux-gnu/libicudata.so.67 (0x00007f5c72527000)
libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f5c724bc000)
libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f5c7241c000)
libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f5c7220e000)
libjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f5c72188000)
libdeflate.so.0 => /usr/lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f5c7216c000)
libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f5c72144000)
libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f5c72125000)
librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f5c71f08000)
libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f5c71eda000)
libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f5c71ec5000)
libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f5c71e8d000)
libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f5c71ce0000)
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f5c71c93000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f5c71bb3000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f5c71b7f000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f5c71b77000)
libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f5c71b23000)
liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f5c71b12000)
libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f5c7198e000)
libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f5c71955000)
libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f5c718d0000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f5c717b2000)
libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f5c71683000)
libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f5c71470000)
libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f5c71461000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f5c71458000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5c7143e000)
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f5c71421000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5c713fe000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f5c713f4000)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804368372,https://api.github.com/repos/simonw/datasette/issues/1249,804368372,MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg==,9599,2021-03-22T20:22:43Z,2021-03-22T20:22:43Z,OWNER,"```dockerfile
FROM python:3.9.2-slim-buster as build
# software-properties-common provides add-apt-repository
RUN apt-get update && \
apt-get -y --no-install-recommends install software-properties-common && \
add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
apt-get update && \
apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg
RUN pip install datasette && \
find /usr/local/lib -name '__pycache__' | xargs rm -r && \
rm -rf /root/.cache/pip
EXPOSE 8001
CMD [""datasette""]
```
262 MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804363687,https://api.github.com/repos/simonw/datasette/issues/1249,804363687,MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw==,9599,2021-03-22T20:15:00Z,2021-03-22T20:15:00Z,OWNER,"```
RUN pip install datasette && \
find /usr/local/lib -name '__pycache__' | xargs rm -r
```
That dropped it to 265MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804360701,https://api.github.com/repos/simonw/datasette/issues/1249,804360701,MDEyOklzc3VlQ29tbWVudDgwNDM2MDcwMQ==,9599,2021-03-22T20:10:07Z,2021-03-22T20:10:07Z,OWNER,Adding `--no-install-recommends` dropped it to 275MB,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804347152,https://api.github.com/repos/simonw/datasette/issues/1249,804347152,MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg==,9599,2021-03-22T19:47:56Z,2021-03-22T19:48:03Z,OWNER,I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804344553,https://api.github.com/repos/simonw/datasette/issues/1249,804344553,MDEyOklzc3VlQ29tbWVudDgwNDM0NDU1Mw==,9599,2021-03-22T19:43:25Z,2021-03-22T19:43:25Z,OWNER,Does `--no-install-recommends` make a difference?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804338678,https://api.github.com/repos/simonw/datasette/issues/1249,804338678,MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA==,9599,2021-03-22T19:33:43Z,2021-03-22T19:33:43Z,OWNER,"Replacing `rm -rf /var/lib/{apt,dpkg,cache,log}/` with
```
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg
```
Got the size down to 305MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804318314,https://api.github.com/repos/simonw/datasette/issues/1249,804318314,MDEyOklzc3VlQ29tbWVudDgwNDMxODMxNA==,9599,2021-03-22T19:04:30Z,2021-03-22T19:04:30Z,OWNER,Considering the image on Docker Hub right now is `383MB` this is actually an improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804317545,https://api.github.com/repos/simonw/datasette/issues/1249,804317545,MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ==,9599,2021-03-22T19:03:22Z,2021-03-22T19:03:22Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build
# software-properties-common provides add-apt-repository
RUN apt-get update && \
apt-get -y install software-properties-common && \
add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
apt-get update && \
apt-get -t sid install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
RUN pip install datasette
EXPOSE 8001
CMD [""datasette""]
```
Produces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804310353,https://api.github.com/repos/simonw/datasette/issues/1249,804310353,MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw==,9599,2021-03-22T18:52:12Z,2021-03-22T18:52:12Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build
# Setup build dependencies
RUN apt update \
&& apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
&& apt clean
RUN wget ""https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz"" && tar xzf sqlite-autoconf-3340100.tar.gz \
&& cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
CFLAGS=""-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1"" \
&& make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz"" && tar zxf freexl-1.0.6.tar.gz \
&& cd freexl-1.0.6 && ./configure && make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz"" && tar zxf libspatialite-5.0.1.tar.gz \
&& cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz"" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz"" && tar zxf spatialite-tools-5.0.0.tar.gz \
&& cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install
# Add local code to the image instead of fetching from pypi.
#COPY . /datasette
#RUN pip install /datasette
RUN pip install datasette
FROM python:3.9.2-slim-buster
# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```
Produced a 448MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-804309510,https://api.github.com/repos/simonw/datasette/issues/1249,804309510,MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA==,9599,2021-03-22T18:50:50Z,2021-03-22T18:50:50Z,OWNER,"Ideally I'd like to use the Debian stable `python:3.9.2-slim-buster` base image but install SpatiaLite from Debian unstable here: https://packages.debian.org/sid/libspatialite7
This pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/pull/1271#issuecomment-804265042,https://api.github.com/repos/simonw/datasette/issues/1271,804265042,MDEyOklzc3VlQ29tbWVudDgwNDI2NTA0Mg==,9599,2021-03-22T17:45:45Z,2021-03-22T17:45:45Z,OWNER,"I can remove this code too:
https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L190-L192","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,
https://github.com/simonw/datasette/issues/1249#issuecomment-804263434,https://api.github.com/repos/simonw/datasette/issues/1249,804263434,MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA==,9599,2021-03-22T17:43:25Z,2021-03-22T17:43:25Z,OWNER,I figured out the cause of the hang in #1268 - it was caused by `select count(*) from SpatialIndex` interacting badly with the `set_progress_handler()` mechanism I was using to implement query time limits. #1271 has a replacement for that using `asyncio.wait_for()` and `conn.interrupt()` which should resolve the SpatiaLite issue too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1268#issuecomment-804261915,https://api.github.com/repos/simonw/datasette/issues/1268,804261915,MDEyOklzc3VlQ29tbWVudDgwNDI2MTkxNQ==,9599,2021-03-22T17:41:12Z,2021-03-22T17:41:12Z,OWNER,"Closing this because I've figured out the root of the problem now, and I have a potential solution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1269#issuecomment-804261610,https://api.github.com/repos/simonw/datasette/issues/1269,804261610,MDEyOklzc3VlQ29tbWVudDgwNDI2MTYxMA==,9599,2021-03-22T17:40:41Z,2021-03-22T17:40:41Z,OWNER,"#1270 looks promising, and I don't want to leave open a security hole where someone could potentially hang Datasette with a nasty `count(*)` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,
https://github.com/simonw/datasette/pull/1271#issuecomment-804261103,https://api.github.com/repos/simonw/datasette/issues/1271,804261103,MDEyOklzc3VlQ29tbWVudDgwNDI2MTEwMw==,22429695,2021-03-22T17:39:57Z,2021-03-22T17:39:57Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=h1) Report
> Merging [#1271](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=desc) (fb2ad7a) into [main](https://codecov.io/gh/simonw/datasette/commit/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b?el=desc) (c4f1ec7) will **decrease** coverage by `0.28%`.
> The diff coverage is `94.28%`.
[](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #1271 +/- ##
==========================================
- Coverage 91.51% 91.22% -0.29%
==========================================
Files 34 34
Lines 4255 4263 +8
==========================================
- Hits 3894 3889 -5
- Misses 361 374 +13
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1271/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.41% <94.28%> (-0.52%)` | :arrow_down: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1271/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `92.24% <0.00%> (-1.90%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=footer). Last update [c4f1ec7...fb2ad7a](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,
https://github.com/simonw/datasette/issues/1270#issuecomment-804255633,https://api.github.com/repos/simonw/datasette/issues/1270,804255633,MDEyOklzc3VlQ29tbWVudDgwNDI1NTYzMw==,9599,2021-03-22T17:32:02Z,2021-03-22T17:32:08Z,OWNER,Confirmed that the `interrupt()` based cancellation mechanism fixes the SpatiaLite issue in #1268!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,
https://github.com/simonw/datasette/issues/1270#issuecomment-803834784,https://api.github.com/repos/simonw/datasette/issues/1270,803834784,MDEyOklzc3VlQ29tbWVudDgwMzgzNDc4NA==,9599,2021-03-22T07:31:57Z,2021-03-22T16:22:19Z,OWNER,"I think the implementation for this goes here: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L146-L157
I figured out a similar pattern in `datasette-ripgrep` here: https://github.com/simonw/datasette-ripgrep/blob/0.7/datasette_ripgrep/__init__.py#L63-L71","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,
https://github.com/simonw/datasette/issues/1268#issuecomment-803802957,https://api.github.com/repos/simonw/datasette/issues/1268,803802957,MDEyOklzc3VlQ29tbWVudDgwMzgwMjk1Nw==,9599,2021-03-22T06:38:14Z,2021-03-22T06:38:14Z,OWNER,"Also worth trying is to change this code:
```python
n = 1000
if ms < 50:
n = 1
```
What happens with `n = 10` instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1269#issuecomment-803785808,https://api.github.com/repos/simonw/datasette/issues/1269,803785808,MDEyOklzc3VlQ29tbWVudDgwMzc4NTgwOA==,9599,2021-03-22T06:00:53Z,2021-03-22T06:00:53Z,OWNER,This may not be necessary if using `.interrupt() for SQLite timeouts in #1270 works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,
https://github.com/simonw/datasette/issues/1268#issuecomment-803784902,https://api.github.com/repos/simonw/datasette/issues/1268,803784902,MDEyOklzc3VlQ29tbWVudDgwMzc4NDkwMg==,9599,2021-03-22T05:59:06Z,2021-03-22T05:59:06Z,OWNER,"Even if I implement that workaround in #1269 I'm concerned that this could still allow users to deliberately crash Datasette (if it's running SpatiaLite 5.0) by executing `select count(*) from SpatialIndex`.
That `interrupt` timeout mechanism is worth digging into further.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803782705,https://api.github.com/repos/simonw/datasette/issues/1268,803782705,MDEyOklzc3VlQ29tbWVudDgwMzc4MjcwNQ==,9599,2021-03-22T05:54:19Z,2021-03-22T05:54:19Z,OWNER,"Got two new TILs out of this:
* [Tracing every executed Python statement](https://til.simonwillison.net/python/tracing-every-statement)
* [Running gdb against a Python process in a running Docker container](https://til.simonwillison.net/docker/gdb-python-docker)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803777724,https://api.github.com/repos/simonw/datasette/issues/1268,803777724,MDEyOklzc3VlQ29tbWVudDgwMzc3NzcyNA==,9599,2021-03-22T05:42:50Z,2021-03-22T05:43:23Z,OWNER,"![]()
If I want to avoid counting virtual tables, I need to detect which tables are virtual tables.
The safest way to do this is probably to pull the `sql` for every table and then, in Python, check for values that start with `create virtual table` after converting to lower case, using any number of spaces.
This would catch things like ` CREATE virtual TABLE` which might be missed by a SQL `like` query. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803775121,https://api.github.com/repos/simonw/datasette/issues/1268,803775121,MDEyOklzc3VlQ29tbWVudDgwMzc3NTEyMQ==,9599,2021-03-22T05:36:26Z,2021-03-22T05:36:26Z,OWNER,So one fix could be to avoid running counts for anything that turns out to be a virtual table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774926,https://api.github.com/repos/simonw/datasette/issues/1268,803774926,MDEyOklzc3VlQ29tbWVudDgwMzc3NDkyNg==,9599,2021-03-22T05:35:56Z,2021-03-22T05:35:56Z,OWNER,That's in this code here: https://github.com/simonw/datasette/blob/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b/datasette/database.py#L221-L241,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774518,https://api.github.com/repos/simonw/datasette/issues/1268,803774518,MDEyOklzc3VlQ29tbWVudDgwMzc3NDUxOA==,9599,2021-03-22T05:34:57Z,2021-03-22T05:34:57Z,OWNER,"... and sure enough, adding this code fixed the problem:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 3579cce..b466b12 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -224,6 +226,9 @@ class Database:
# Try to get counts for each table, $limit timeout for each count
counts = {}
for table in await self.table_names():
+ if table == ""SpatialIndex"":
+ counts[table] = 0
+ continue
try:
table_count = (
await self.execute(
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803773484,https://api.github.com/repos/simonw/datasette/issues/1268,803773484,MDEyOklzc3VlQ29tbWVudDgwMzc3MzQ4NA==,9599,2021-03-22T05:32:29Z,2021-03-22T05:32:29Z,OWNER,"To figure out which SQL query triggers the problem I added this code to write to a log file:
```python
with sqlite_timelimit(conn, time_limit_ms):
try:
cursor = conn.cursor()
with open(""/tmp/sql.log"", ""ab"", buffering=0) as fp:
fp.write((""{}: {}\n"".format(sql, params)).encode(""utf-8""))
cursor.execute(sql, params if params is not None else {})
```
I had to use `ab` binary mode because Python doesn't allow `buffering=0` for non-binary file operations.
With the log enabled, I used `docker exec -it 589ae68de943 bash` to attach to the running container and `tail -f /tmp/sql.log` to see the logs. Here's where it broke:
```
select count(*) from [idx_civici_geom_parent]: None
select count(*) from [sqlite_stat1]: None
select count(*) from [sqlite_stat3]: None
select count(*) from [SpatialIndex]: None
```
So attempting to run a `count(*)` against the `SpatialIndex` virtual table is the thing that triggers the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764919,https://api.github.com/repos/simonw/datasette/issues/1268,803764919,MDEyOklzc3VlQ29tbWVudDgwMzc2NDkxOQ==,9599,2021-03-22T05:11:11Z,2021-03-22T05:11:11Z,OWNER,"Maybe I could implement SQLite query timeouts using the `interrupt()` method instead of the progress handler hack I'm currently using?
https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764200,https://api.github.com/repos/simonw/datasette/issues/1268,803764200,MDEyOklzc3VlQ29tbWVudDgwMzc2NDIwMA==,9599,2021-03-22T05:09:13Z,2021-03-22T05:09:13Z,OWNER,"I tried building a container where the `conn.set_progress_handler(handler, n)` line was commented out... and it fixed the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762969,https://api.github.com/repos/simonw/datasette/issues/1268,803762969,MDEyOklzc3VlQ29tbWVudDgwMzc2Mjk2OQ==,9599,2021-03-22T05:05:51Z,2021-03-22T05:05:51Z,OWNER,I had to run `docker kill 16197781a7b5` to kill the broken container - Ctrl+C in the Datasette console window didn't do anything.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762609,https://api.github.com/repos/simonw/datasette/issues/1268,803762609,MDEyOklzc3VlQ29tbWVudDgwMzc2MjYwOQ==,9599,2021-03-22T05:05:00Z,2021-03-22T05:05:00Z,OWNER,"Using https://til.simonwillison.net/docker/attach-bash-to-running-container - I figured out how to run `gdb`. I had to use `--privileged` here because otherwise `gdb` showed a ""Could not attach to process"" error.
```
docker exec --privileged -it 16197781a7b5 bash
# apt-get install gdb python3-dbg
# gdb /usr/bin/python3 -p 20
```
This paused the process. I tried running this:
```
(gdb) py-bt
Traceback (most recent call first):
File ""/usr/lib/python3.8/asyncio/base_events.py"", line 1845, in _run_once
if handle._cancelled:
File ""/usr/lib/python3.8/asyncio/base_events.py"", line 570, in run_forever
self._run_once()
File ""/usr/lib/python3.8/asyncio/base_events.py"", line 603, in run_until_complete
self.run_forever()
File ""/usr/local/lib/python3.8/dist-packages/uvicorn/server.py"", line 49, in run
loop.run_until_complete(self.serve(sockets=sockets))
File ""/usr/local/lib/python3.8/dist-packages/uvicorn/main.py"", line 386, in run
server.run()
File ""/usr/local/lib/python3.8/dist-packages/datasette/cli.py"", line 575, in serve
uvicorn.run(ds.app(), **uvicorn_kwargs)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/usr/local/bin/datasette"", line 8, in
sys.exit(cli())
File ""/usr/lib/python3.8/trace.py"", line 450, in runctx
exec(cmd, globals, locals)
File ""/usr/lib/python3.8/trace.py"", line 6632, in main
File ""/usr/lib/python3.8/trace.py"", line 756, in
main()
File ""/usr/lib/python3.8/runpy.py"", line 343, in _run_code
File ""/usr/lib/python3.8/runpy.py"", line 450, in _run_module_as_main
```
Not sure if that's useful or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803759051,https://api.github.com/repos/simonw/datasette/issues/1268,803759051,MDEyOklzc3VlQ29tbWVudDgwMzc1OTA1MQ==,9599,2021-03-22T04:55:22Z,2021-03-22T04:55:22Z,OWNER,So I think there's a bug in the way the `set_progress_handler()` mechanism works when used in conjunction with SpatiaLite 5.0 on Linux.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803758793,https://api.github.com/repos/simonw/datasette/issues/1268,803758793,MDEyOklzc3VlQ29tbWVudDgwMzc1ODc5Mw==,9599,2021-03-22T04:54:32Z,2021-03-22T04:54:32Z,OWNER,"Hitting http://localhost:8001/tuscany_housenumbers triggers the bug. It gets stuck in a loop that looks like this:
![]()
Which looks to me like this code: https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L139-L158","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803758182,https://api.github.com/repos/simonw/datasette/issues/1268,803758182,MDEyOklzc3VlQ29tbWVudDgwMzc1ODE4Mg==,9599,2021-03-22T04:52:15Z,2021-03-22T04:52:15Z,OWNER,Hitting http://localhost:8001/ successfully shows the homepage (after a lot more scrolling).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803757746,https://api.github.com/repos/simonw/datasette/issues/1268,803757746,MDEyOklzc3VlQ29tbWVudDgwMzc1Nzc0Ng==,9599,2021-03-22T04:50:40Z,2021-03-22T04:51:52Z,OWNER,"Here's a fun debugging trick:
docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
root@16197781a7b5:/# python3 -m trace --trace $(which datasette) \
-p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite \
--load-extension=spatialite
A huge amount of stuff scrolls past as Datasette starts up, since we are tracing every executed line of Python.
After about a minute it's finished starting and gets to this point:
```
selectors.py(452): if timeout is None:
selectors.py(454): elif timeout <= 0:
selectors.py(459): timeout = math.ceil(timeout * 1e3) * 1e-3
selectors.py(464): max_ev = max(len(self._fd_to_key), 1)
selectors.py(466): ready = []
selectors.py(467): try:
selectors.py(468): fd_event_list = self._selector.poll(timeout, max_ev)
```
Now I can make some HTTP requests against it.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1268#issuecomment-803756495,https://api.github.com/repos/simonw/datasette/issues/1268,803756495,MDEyOklzc3VlQ29tbWVudDgwMzc1NjQ5NQ==,9599,2021-03-22T04:46:04Z,2021-03-22T04:46:04Z,OWNER,`gdb` may be able to help debug this: https://www.podoliaka.org/2016/04/10/debugging-cpython-gdb/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,
https://github.com/simonw/datasette/issues/1249#issuecomment-803755698,https://api.github.com/repos/simonw/datasette/issues/1249,803755698,MDEyOklzc3VlQ29tbWVudDgwMzc1NTY5OA==,9599,2021-03-22T04:43:02Z,2021-03-22T04:43:02Z,OWNER,I'll spin off a separate ticket to investigate the hang.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1267#issuecomment-803754226,https://api.github.com/repos/simonw/datasette/issues/1267,803754226,MDEyOklzc3VlQ29tbWVudDgwMzc1NDIyNg==,9599,2021-03-22T04:37:26Z,2021-03-22T04:37:26Z,OWNER,"Thanks for doing this - I've used alternativeto.net a bunch in the past, it's great to see Datasette listed there.
This does raise some interesting philosophical questions: three years into the project I'm still not entirely sure what Datasette competes with! Could be SQLite desktop packages, could be visualization software like Tableau, could even be something like Airtable (given a few more plugins).
It will be interesting to see how the alternativeto listing evolves, maybe it will help me answer that question!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837208901,
https://github.com/simonw/datasette/issues/1249#issuecomment-803753388,https://api.github.com/repos/simonw/datasette/issues/1249,803753388,MDEyOklzc3VlQ29tbWVudDgwMzc1MzM4OA==,9599,2021-03-22T04:34:20Z,2021-03-22T04:35:10Z,OWNER,"Well this is frustrating. I finally found a Dockerfile that worked and installed an Ubuntu pre-compiled SpatiaLite module that would load...
```dockerfile
FROM ubuntu:20.10 as install_spatialite
RUN apt update && \
apt install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
FROM ubuntu:20.10
RUN apt update && \
apt install -y python3-pip && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
RUN pip install datasette
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```
(Which produced a 550MB image)
And when I ran Datasette I got that same error where the database listing page hangs!
```
docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite --load-extension=spatialite
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803751068,https://api.github.com/repos/simonw/datasette/issues/1249,803751068,MDEyOklzc3VlQ29tbWVudDgwMzc1MTA2OA==,9599,2021-03-22T04:26:45Z,2021-03-22T04:26:45Z,OWNER,"Here's why:
```
datasette % docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
root@3430352ff378:/# datasette
bash: /usr/local/bin/datasette: /usr/bin/python3: bad interpreter: No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803750617,https://api.github.com/repos/simonw/datasette/issues/1249,803750617,MDEyOklzc3VlQ29tbWVudDgwMzc1MDYxNw==,9599,2021-03-22T04:25:14Z,2021-03-22T04:25:14Z,OWNER,"Got this error attempting to run Datasette (with or without SpatiaLite):
```
standard_init_linux.go:219: exec user process caused: no such file or directory
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803750399,https://api.github.com/repos/simonw/datasette/issues/1249,803750399,MDEyOklzc3VlQ29tbWVudDgwMzc1MDM5OQ==,9599,2021-03-22T04:24:25Z,2021-03-22T04:24:25Z,OWNER,"I'll try using `ubuntu:20.10` for everything:
```dockerfile
FROM ubuntu:20.10 as install_spatialite
RUN apt update && \
apt install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
FROM ubuntu:20.10 as build
RUN apt update && \
apt install -y python3-pip && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
RUN pip install datasette
#COPY . /datasette
#RUN pip install /datasette
FROM ubuntu:20.10
# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803749831,https://api.github.com/repos/simonw/datasette/issues/1249,803749831,MDEyOklzc3VlQ29tbWVudDgwMzc0OTgzMQ==,9599,2021-03-22T04:22:35Z,2021-03-22T04:22:35Z,OWNER,"I tried copying just the `mod_spatialite.so` file:
```dockerfile
FROM ubuntu:20.10 as install_spatialite
RUN apt update && \
apt install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
FROM python:3.9.2-slim as build
RUN pip install datasette
#COPY . /datasette
#RUN pip install /datasette
FROM python:3.9.2-slim
# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```
But when I ran Datasette with `--load-extension=spatialite` I got this:
```
File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 151, in in_thread
self.ds._prepare_connection(conn, self.name)
File ""/usr/local/lib/python3.9/site-packages/datasette/app.py"", line 502, in _prepare_connection
conn.execute(f""SELECT load_extension('{extension}')"")
sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803748469,https://api.github.com/repos/simonw/datasette/issues/1249,803748469,MDEyOklzc3VlQ29tbWVudDgwMzc0ODQ2OQ==,9599,2021-03-22T04:17:51Z,2021-03-22T04:17:51Z,OWNER,"... except my clever image using SpatiaLite installed for Ubuntu doesn't actually work:
```
datasette % docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
File ""/usr/local/lib/python3.9/sqlite3/dbapi2.py"", line 27, in
from _sqlite3 import *
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /usr/lib/x86_64-linux-gnu/libsqlite3.so.0)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803748158,https://api.github.com/repos/simonw/datasette/issues/1249,803748158,MDEyOklzc3VlQ29tbWVudDgwMzc0ODE1OA==,9599,2021-03-22T04:16:57Z,2021-03-22T04:16:57Z,OWNER,"Which is great, because the image on Docker Hub right now is 383MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803747701,https://api.github.com/repos/simonw/datasette/issues/1249,803747701,MDEyOklzc3VlQ29tbWVudDgwMzc0NzcwMQ==,9599,2021-03-22T04:15:40Z,2021-03-22T04:15:40Z,OWNER,"Here's a trick: install SpatiaLite in `ubuntu:20.10` and then copy it into the final `python:3.9.2-slim` image.
```dockerfile
FROM ubuntu:20.10 as install_spatialite
RUN apt update && \
apt install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
FROM python:3.9.2-slim as build
RUN pip install datasette
#COPY . /datasette
#RUN pip install /datasette
FROM python:3.9.2-slim
# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```
That produced a 265MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803700940,https://api.github.com/repos/simonw/datasette/issues/1249,803700940,MDEyOklzc3VlQ29tbWVudDgwMzcwMDk0MA==,9599,2021-03-22T01:14:24Z,2021-03-22T01:14:24Z,OWNER,I tried that with just `python3-pip` (removing `libsqlite3-mod-spatialite`) and got 435MB.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803700626,https://api.github.com/repos/simonw/datasette/issues/1249,803700626,MDEyOklzc3VlQ29tbWVudDgwMzcwMDYyNg==,9599,2021-03-22T01:13:04Z,2021-03-22T01:13:04Z,OWNER,"Building a Dockerfile containing just `FROM ubuntu:20.10` gave me `79.5MB`.
Building this one:
```dockerfile
FROM ubuntu:20.10
# Setup build dependencies
RUN apt update && \
apt install -y python3-pip libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
```
Resulted in a 515MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803698983,https://api.github.com/repos/simonw/datasette/issues/1249,803698983,MDEyOklzc3VlQ29tbWVudDgwMzY5ODk4Mw==,9599,2021-03-22T01:05:36Z,2021-03-22T01:06:23Z,OWNER,"It's pretty big though. I tried this version which avoids copying junk from my laptop in:
```dockerfile
FROM ubuntu:20.10
# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean
RUN pip install datasette
EXPOSE 8001
CMD [""datasette""]
```
And got this:
```
datasette % docker images datasette-spatialite
REPOSITORY TAG IMAGE ID CREATED SIZE
datasette-spatialite latest 0796950653c2 2 seconds ago 528MB
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803698168,https://api.github.com/repos/simonw/datasette/issues/1249,803698168,MDEyOklzc3VlQ29tbWVudDgwMzY5ODE2OA==,9599,2021-03-22T01:02:02Z,2021-03-22T01:02:30Z,OWNER,"This is the shortest Dockerfile that appeared to give me a working SpatiaLite module:
```dockerfile
FROM ubuntu:20.10
# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean
# Add local code to the image instead of fetching from pypi.
COPY . /datasette
RUN pip install /datasette
RUN rm -rf /datasette
EXPOSE 8001
CMD [""datasette""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,
https://github.com/simonw/datasette/issues/1249#issuecomment-803697546,https://api.github.com/repos/simonw/datasette/issues/1249,803697546,MDEyOklzc3VlQ29tbWVudDgwMzY5NzU0Ng==,9599,2021-03-22T00:59:47Z,2021-03-22T00:59:47Z,OWNER,"To debug I'm running:
docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
This gets me a shell I can use.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,