45 rows where issue = 824064069 sorted by updated_at descending

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

user

issue

  • Updated Dockerfile with SpatiaLite version 5.0 · 45

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
808998719 https://github.com/simonw/datasette/issues/1249#issuecomment-808998719 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwODk5ODcxOQ== simonw 9599 2021-03-29T00:57:13Z 2021-03-29T00:57:13Z OWNER

I just shipped Datasette 0.56 - here's the CI run: https://github.com/simonw/datasette/runs/2214701802?check_suite_focus=true

It pushed a new latest tag to https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated

docker pull datasetteproject/datasette:latest

And then:

docker run datasetteproject/datasette:latest datasette \
  --load-extension=spatialite \
  --get /-/versions.json | jq .sqlite

Outputs:

{
  "version": "3.27.2",
  "fts_versions": [
    "FTS5",
    "FTS4",
    "FTS3"
  ],
  "extensions": {
    "json1": null,
    "spatialite": "5.0.1"
  },
  "compile_options": [
    "COMPILER=gcc-8.3.0",
    "ENABLE_COLUMN_METADATA",
    "ENABLE_DBSTAT_VTAB",
    "ENABLE_FTS3",
    "ENABLE_FTS3_PARENTHESIS",
    "ENABLE_FTS3_TOKENIZER",
    "ENABLE_FTS4",
    "ENABLE_FTS5",
    "ENABLE_JSON1",
    "ENABLE_LOAD_EXTENSION",
    "ENABLE_PREUPDATE_HOOK",
    "ENABLE_RTREE",
    "ENABLE_SESSION",
    "ENABLE_STMTVTAB",
    "ENABLE_UNLOCK_NOTIFY",
    "ENABLE_UPDATE_DELETE_LIMIT",
    "HAVE_ISNAN",
    "LIKE_DOESNT_MATCH_BLOBS",
    "MAX_SCHEMA_RETRY=25",
    "MAX_VARIABLE_NUMBER=250000",
    "OMIT_LOOKASIDE",
    "SECURE_DELETE",
    "SOUNDEX",
    "TEMP_STORE=1",
    "THREADSAFE=1",
    "USE_URI"
  ]
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
808649480 https://github.com/simonw/datasette/issues/1249#issuecomment-808649480 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwODY0OTQ4MA== simonw 9599 2021-03-27T04:32:10Z 2021-03-27T04:32:10Z OWNER

I'll close this issue after I ship Datasette 0.56 and confirm that the Dockerfile was correctly built and published to Docker Hub.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
808649322 https://github.com/simonw/datasette/issues/1249#issuecomment-808649322 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwODY0OTMyMg== simonw 9599 2021-03-27T04:31:28Z 2021-03-27T04:31:28Z OWNER

One last test of that Dockerfile:

(datasette) datasette % docker build -f Dockerfile -t datasetteproject/datasette:0.55a --build-arg VERSION=0.55 .
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' | jq
{
  "python": {
    "version": "3.9.2",
    "full": "3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"
  },
  "datasette": {
    "version": "0.55"
  },
  "asgi": "3.0",
  "uvicorn": "0.13.4",
  "sqlite": {
    "version": "3.27.2",
    "fts_versions": [
      "FTS5",
      "FTS4",
      "FTS3"
    ],
    "extensions": {
      "json1": null
    },
    "compile_options": [
      "COMPILER=gcc-8.3.0",
      "ENABLE_COLUMN_METADATA",
      "ENABLE_DBSTAT_VTAB",
      "ENABLE_FTS3",
      "ENABLE_FTS3_PARENTHESIS",
      "ENABLE_FTS3_TOKENIZER",
      "ENABLE_FTS4",
      "ENABLE_FTS5",
      "ENABLE_JSON1",
      "ENABLE_LOAD_EXTENSION",
      "ENABLE_PREUPDATE_HOOK",
      "ENABLE_RTREE",
      "ENABLE_SESSION",
      "ENABLE_STMTVTAB",
      "ENABLE_UNLOCK_NOTIFY",
      "ENABLE_UPDATE_DELETE_LIMIT",
      "HAVE_ISNAN",
      "LIKE_DOESNT_MATCH_BLOBS",
      "MAX_SCHEMA_RETRY=25",
      "MAX_VARIABLE_NUMBER=250000",
      "OMIT_LOOKASIDE",
      "SECURE_DELETE",
      "SOUNDEX",
      "TEMP_STORE=1",
      "THREADSAFE=1",
      "USE_URI"
    ]
  }
}
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' --load-extension=spatialite | jq
{
  "python": {
    "version": "3.9.2",
    "full": "3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"
  },
  "datasette": {
    "version": "0.55"
  },
  "asgi": "3.0",
  "uvicorn": "0.13.4",
  "sqlite": {
    "version": "3.27.2",
    "fts_versions": [
      "FTS5",
      "FTS4",
      "FTS3"
    ],
    "extensions": {
      "json1": null,
      "spatialite": "5.0.1"
    },
    "compile_options": [
      "COMPILER=gcc-8.3.0",
      "ENABLE_COLUMN_METADATA",
      "ENABLE_DBSTAT_VTAB",
      "ENABLE_FTS3",
      "ENABLE_FTS3_PARENTHESIS",
      "ENABLE_FTS3_TOKENIZER",
      "ENABLE_FTS4",
      "ENABLE_FTS5",
      "ENABLE_JSON1",
      "ENABLE_LOAD_EXTENSION",
      "ENABLE_PREUPDATE_HOOK",
      "ENABLE_RTREE",
      "ENABLE_SESSION",
      "ENABLE_STMTVTAB",
      "ENABLE_UNLOCK_NOTIFY",
      "ENABLE_UPDATE_DELETE_LIMIT",
      "HAVE_ISNAN",
      "LIKE_DOESNT_MATCH_BLOBS",
      "MAX_SCHEMA_RETRY=25",
      "MAX_VARIABLE_NUMBER=250000",
      "OMIT_LOOKASIDE",
      "SECURE_DELETE",
      "SOUNDEX",
      "TEMP_STORE=1",
      "THREADSAFE=1",
      "USE_URI"
    ]
  }
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
805033155 https://github.com/simonw/datasette/issues/1249#issuecomment-805033155 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNTAzMzE1NQ== simonw 9599 2021-03-23T16:12:13Z 2021-03-23T16:12:13Z OWNER

Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source

The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful VirtualKNN module), take a look at the Datasette Dockerfile.

See also #1273

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804406675 https://github.com/simonw/datasette/issues/1249#issuecomment-804406675 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDQwNjY3NQ== simonw 9599 2021-03-22T21:26:27Z 2021-03-22T21:26:27Z OWNER

(Without the apt-get update ... SpatiaLite line it's 125MB)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804404544 https://github.com/simonw/datasette/issues/1249#issuecomment-804404544 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA== simonw 9599 2021-03-22T21:22:56Z 2021-03-22T21:24:24Z OWNER

Final version of Dockerfile which installs the specified version from GitHub:

docker build . -t datasette-spatialite --build-arg VERSION=0.55
FROM python:3.9.2-slim-buster as build

# Version of Datasette to install, e.g. 0.55
#   docker build . -t datasette --build-arg VERSION=0.55
ARG VERSION

# software-properties-common provides add-apt-repository
# which we need in order to install a more recent release
# of libsqlite3-mod-spatialite from the sid distribution
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt-get remove -y software-properties-common && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD ["datasette"]

Run against 0.55 this produces an image of 262MB

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804384196 https://github.com/simonw/datasette/issues/1249#issuecomment-804384196 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng== simonw 9599 2021-03-22T20:48:46Z 2021-03-22T20:48:46Z OWNER

I think part of the reason it's smaller is that I ran pip install datasette instead of using COPY . /datasette followed by pip install /datasette.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804380181 https://github.com/simonw/datasette/issues/1249#issuecomment-804380181 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ== simonw 9599 2021-03-22T20:42:16Z 2021-03-22T20:42:16Z OWNER

Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804379644 https://github.com/simonw/datasette/issues/1249#issuecomment-804379644 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA== simonw 9599 2021-03-22T20:41:23Z 2021-03-22T20:41:23Z OWNER

I tried adding apt-get remove -y software-properties-common && to remove software-properties-common but it made no difference to the image size.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804372977 https://github.com/simonw/datasette/issues/1249#issuecomment-804372977 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw== simonw 9599 2021-03-22T20:30:37Z 2021-03-22T20:30:37Z OWNER

I tried copying just the mod_spatialite.so file into a second stage build but it failed. So I ran bash in a working image and used ldd to figure out what it was linked to:

root@39683f91e588:/usr/lib/x86_64-linux-gnu# ldd mod_spatialite.so
    linux-vdso.so.1 (0x00007ffd021f4000)
    libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f5c75412000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c753f0000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c752ac000)
    libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f5c750a0000)
    librttopo.so.1 => /usr/lib/x86_64-linux-gnu/librttopo.so.1 (0x00007f5c75028000)
    libfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f5c7501c000)
    libproj.so.19 => /usr/lib/x86_64-linux-gnu/libproj.so.19 (0x00007f5c74ca7000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c74a89000)
    libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f5c74967000)
    libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f5c7492b000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c74766000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c74760000)
    libicuuc.so.67 => /usr/lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007f5c74575000)
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5c7454d000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f5c75d49000)
    libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f5c744c7000)
    libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f5c74439000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5c7426c000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c74250000)
    libgeos-3.9.0.so => /usr/lib/x86_64-linux-gnu/libgeos-3.9.0.so (0x00007f5c74040000)
    libicudata.so.67 => /usr/lib/x86_64-linux-gnu/libicudata.so.67 (0x00007f5c72527000)
    libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f5c724bc000)
    libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f5c7241c000)
    libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f5c7220e000)
    libjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f5c72188000)
    libdeflate.so.0 => /usr/lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f5c7216c000)
    libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f5c72144000)
    libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f5c72125000)
    librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f5c71f08000)
    libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f5c71eda000)
    libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f5c71ec5000)
    libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f5c71e8d000)
    libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f5c71ce0000)
    libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f5c71c93000)
    libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f5c71bb3000)
    libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f5c71b7f000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f5c71b77000)
    libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f5c71b23000)
    liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f5c71b12000)
    libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f5c7198e000)
    libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f5c71955000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f5c718d0000)
    libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f5c717b2000)
    libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f5c71683000)
    libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f5c71470000)
    libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f5c71461000)
    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f5c71458000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5c7143e000)
    libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f5c71421000)
    libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5c713fe000)
    libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f5c713f4000)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804368372 https://github.com/simonw/datasette/issues/1249#issuecomment-804368372 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg== simonw 9599 2021-03-22T20:22:43Z 2021-03-22T20:22:43Z OWNER
FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD ["datasette"]

262 MB

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804363687 https://github.com/simonw/datasette/issues/1249#issuecomment-804363687 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw== simonw 9599 2021-03-22T20:15:00Z 2021-03-22T20:15:00Z OWNER
RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r

That dropped it to 265MB.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804360701 https://github.com/simonw/datasette/issues/1249#issuecomment-804360701 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM2MDcwMQ== simonw 9599 2021-03-22T20:10:07Z 2021-03-22T20:10:07Z OWNER

Adding --no-install-recommends dropped it to 275MB

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804347152 https://github.com/simonw/datasette/issues/1249#issuecomment-804347152 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg== simonw 9599 2021-03-22T19:47:56Z 2021-03-22T19:48:03Z OWNER

I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804344553 https://github.com/simonw/datasette/issues/1249#issuecomment-804344553 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM0NDU1Mw== simonw 9599 2021-03-22T19:43:25Z 2021-03-22T19:43:25Z OWNER

Does --no-install-recommends make a difference?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804338678 https://github.com/simonw/datasette/issues/1249#issuecomment-804338678 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA== simonw 9599 2021-03-22T19:33:43Z 2021-03-22T19:33:43Z OWNER

Replacing rm -rf /var/lib/{apt,dpkg,cache,log}/ with

    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

Got the size down to 305MB.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804318314 https://github.com/simonw/datasette/issues/1249#issuecomment-804318314 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDMxODMxNA== simonw 9599 2021-03-22T19:04:30Z 2021-03-22T19:04:30Z OWNER

Considering the image on Docker Hub right now is 383MB this is actually an improvement.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804317545 https://github.com/simonw/datasette/issues/1249#issuecomment-804317545 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ== simonw 9599 2021-03-22T19:03:22Z 2021-03-22T19:03:22Z OWNER

This Dockerfile:

FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y install software-properties-common && \
    add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
    apt-get update && \
    apt-get -t sid install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

EXPOSE 8001
CMD ["datasette"]

Produces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804310353 https://github.com/simonw/datasette/issues/1249#issuecomment-804310353 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw== simonw 9599 2021-03-22T18:52:12Z 2021-03-22T18:52:12Z OWNER

This Dockerfile:

FROM python:3.9.2-slim-buster as build

# Setup build dependencies
RUN apt update \
    && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
    libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
    && apt clean

RUN wget "https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz" && tar xzf sqlite-autoconf-3340100.tar.gz \
    && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
    CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1" \
    && make && make install

RUN wget "http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz" && tar zxf freexl-1.0.6.tar.gz \
    && cd freexl-1.0.6 && ./configure && make && make install

RUN wget "http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz" && tar zxf libspatialite-5.0.1.tar.gz \
    && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install

RUN wget "http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install

RUN wget "http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz" && tar zxf spatialite-tools-5.0.0.tar.gz \
    && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install

# Add local code to the image instead of fetching from pypi.
#COPY . /datasette
#RUN pip install /datasette
RUN pip install datasette

FROM python:3.9.2-slim-buster

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD ["datasette"]

Produced a 448MB image.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804309510 https://github.com/simonw/datasette/issues/1249#issuecomment-804309510 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA== simonw 9599 2021-03-22T18:50:50Z 2021-03-22T18:50:50Z OWNER

Ideally I'd like to use the Debian stable python:3.9.2-slim-buster base image but install SpatiaLite from Debian unstable here: https://packages.debian.org/sid/libspatialite7

This pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804263434 https://github.com/simonw/datasette/issues/1249#issuecomment-804263434 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA== simonw 9599 2021-03-22T17:43:25Z 2021-03-22T17:43:25Z OWNER

I figured out the cause of the hang in #1268 - it was caused by select count(*) from SpatialIndex interacting badly with the set_progress_handler() mechanism I was using to implement query time limits. #1271 has a replacement for that using asyncio.wait_for() and conn.interrupt() which should resolve the SpatiaLite issue too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803755698 https://github.com/simonw/datasette/issues/1249#issuecomment-803755698 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc1NTY5OA== simonw 9599 2021-03-22T04:43:02Z 2021-03-22T04:43:02Z OWNER

I'll spin off a separate ticket to investigate the hang.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803753388 https://github.com/simonw/datasette/issues/1249#issuecomment-803753388 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc1MzM4OA== simonw 9599 2021-03-22T04:34:20Z 2021-03-22T04:35:10Z OWNER

Well this is frustrating. I finally found a Dockerfile that worked and installed an Ubuntu pre-compiled SpatiaLite module that would load...

FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM ubuntu:20.10

RUN apt update && \
    apt install -y python3-pip && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD ["datasette"]

(Which produced a 550MB image)

And when I ran Datasette I got that same error where the database listing page hangs!

docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite --load-extension=spatialite
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803751068 https://github.com/simonw/datasette/issues/1249#issuecomment-803751068 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc1MTA2OA== simonw 9599 2021-03-22T04:26:45Z 2021-03-22T04:26:45Z OWNER

Here's why:

datasette % docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash                                                      
root@3430352ff378:/# datasette
bash: /usr/local/bin/datasette: /usr/bin/python3: bad interpreter: No such file or directory
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803750617 https://github.com/simonw/datasette/issues/1249#issuecomment-803750617 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc1MDYxNw== simonw 9599 2021-03-22T04:25:14Z 2021-03-22T04:25:14Z OWNER

Got this error attempting to run Datasette (with or without SpatiaLite):

standard_init_linux.go:219: exec user process caused: no such file or directory
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803750399 https://github.com/simonw/datasette/issues/1249#issuecomment-803750399 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc1MDM5OQ== simonw 9599 2021-03-22T04:24:25Z 2021-03-22T04:24:25Z OWNER

I'll try using ubuntu:20.10 for everything:

FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM ubuntu:20.10 as build

RUN apt update && \
    apt install -y python3-pip && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM ubuntu:20.10

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD ["datasette"]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803749831 https://github.com/simonw/datasette/issues/1249#issuecomment-803749831 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc0OTgzMQ== simonw 9599 2021-03-22T04:22:35Z 2021-03-22T04:22:35Z OWNER

I tried copying just the mod_spatialite.so file:

FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM python:3.9.2-slim as build

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM python:3.9.2-slim

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD ["datasette"]

But when I ran Datasette with --load-extension=spatialite I got this:

  File "/usr/local/lib/python3.9/site-packages/datasette/database.py", line 151, in in_thread
    self.ds._prepare_connection(conn, self.name)
  File "/usr/local/lib/python3.9/site-packages/datasette/app.py", line 502, in _prepare_connection
    conn.execute(f"SELECT load_extension('{extension}')")
sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803748469 https://github.com/simonw/datasette/issues/1249#issuecomment-803748469 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc0ODQ2OQ== simonw 9599 2021-03-22T04:17:51Z 2021-03-22T04:17:51Z OWNER

... except my clever image using SpatiaLite installed for Ubuntu doesn't actually work:

datasette % docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
  File "/usr/local/lib/python3.9/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /usr/lib/x86_64-linux-gnu/libsqlite3.so.0)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803748158 https://github.com/simonw/datasette/issues/1249#issuecomment-803748158 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc0ODE1OA== simonw 9599 2021-03-22T04:16:57Z 2021-03-22T04:16:57Z OWNER

Which is great, because the image on Docker Hub right now is 383MB.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803747701 https://github.com/simonw/datasette/issues/1249#issuecomment-803747701 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc0NzcwMQ== simonw 9599 2021-03-22T04:15:40Z 2021-03-22T04:15:40Z OWNER

Here's a trick: install SpatiaLite in ubuntu:20.10 and then copy it into the final python:3.9.2-slim image.

FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM python:3.9.2-slim as build

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM python:3.9.2-slim

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD ["datasette"]

That produced a 265MB image.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803700940 https://github.com/simonw/datasette/issues/1249#issuecomment-803700940 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzcwMDk0MA== simonw 9599 2021-03-22T01:14:24Z 2021-03-22T01:14:24Z OWNER

I tried that with just python3-pip (removing libsqlite3-mod-spatialite) and got 435MB.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803700626 https://github.com/simonw/datasette/issues/1249#issuecomment-803700626 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzcwMDYyNg== simonw 9599 2021-03-22T01:13:04Z 2021-03-22T01:13:04Z OWNER

Building a Dockerfile containing just FROM ubuntu:20.10 gave me 79.5MB.

Building this one:

FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && \
    apt install -y python3-pip libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

Resulted in a 515MB image.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803698983 https://github.com/simonw/datasette/issues/1249#issuecomment-803698983 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5ODk4Mw== simonw 9599 2021-03-22T01:05:36Z 2021-03-22T01:06:23Z OWNER

It's pretty big though. I tried this version which avoids copying junk from my laptop in:

FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

RUN pip install datasette

EXPOSE 8001
CMD ["datasette"]

And got this:

datasette % docker images datasette-spatialite    
REPOSITORY             TAG       IMAGE ID       CREATED         SIZE
datasette-spatialite   latest    0796950653c2   2 seconds ago   528MB
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803698168 https://github.com/simonw/datasette/issues/1249#issuecomment-803698168 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5ODE2OA== simonw 9599 2021-03-22T01:02:02Z 2021-03-22T01:02:30Z OWNER

This is the shortest Dockerfile that appeared to give me a working SpatiaLite module:

FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

# Add local code to the image instead of fetching from pypi.
COPY . /datasette

RUN pip install /datasette

RUN rm -rf /datasette

EXPOSE 8001
CMD ["datasette"]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803697546 https://github.com/simonw/datasette/issues/1249#issuecomment-803697546 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5NzU0Ng== simonw 9599 2021-03-22T00:59:47Z 2021-03-22T00:59:47Z OWNER

To debug I'm running:

docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash

This gets me a shell I can use.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803697211 https://github.com/simonw/datasette/issues/1249#issuecomment-803697211 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5NzIxMQ== simonw 9599 2021-03-22T00:58:01Z 2021-03-22T00:58:01Z OWNER

I'm messing around with the Dockerfile and after each change I'm running:

docker build . -t datasette-spatialite

And then:

docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803694661 https://github.com/simonw/datasette/issues/1249#issuecomment-803694661 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5NDY2MQ== simonw 9599 2021-03-22T00:46:49Z 2021-03-22T00:46:49Z OWNER

Actually for the loadable module I think I need https://packages.ubuntu.com/groovy/libsqlite3-mod-spatialite

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803694436 https://github.com/simonw/datasette/issues/1249#issuecomment-803694436 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5NDQzNg== simonw 9599 2021-03-22T00:46:00Z 2021-03-22T00:46:00Z OWNER

So I'm going to try 20.10 and see where that gets me.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803694359 https://github.com/simonw/datasette/issues/1249#issuecomment-803694359 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5NDM1OQ== simonw 9599 2021-03-22T00:45:47Z 2021-03-22T00:45:47Z OWNER

https://pythonspeed.com/articles/base-image-python-docker-images/ suggests using python:3.9-slim-buster or ubuntu:20.04 - but 20.04 is focal which still has SpatiaLite 4.3.0a-6build1 - It's 20.10 that has 5.0: https://packages.ubuntu.com/groovy/libspatialite-dev

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803693181 https://github.com/simonw/datasette/issues/1249#issuecomment-803693181 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5MzE4MQ== simonw 9599 2021-03-22T00:41:02Z 2021-03-22T00:41:02Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803692673 https://github.com/simonw/datasette/issues/1249#issuecomment-803692673 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5MjY3Mw== simonw 9599 2021-03-22T00:38:42Z 2021-03-22T00:38:42Z OWNER

Ubuntu Groovy has a package for SpatiaLite 5 - I could try using that instead: https://packages.ubuntu.com/groovy/libspatialite-dev

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803691236 https://github.com/simonw/datasette/issues/1249#issuecomment-803691236 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5MTIzNg== simonw 9599 2021-03-22T00:32:03Z 2021-03-22T00:32:03Z OWNER

Here's something odd: when I run datasette tuscany_housenumbers.sqlite --load-extension=spatialite on macOS against SpatiaLite installed using Homebrew (which reports "spatialite": "5.0.0" on the /-/versions page) I don't get any weird errors at all, everything works fine.

But when I tried compiling SpatiaLite inside the Docker container I had hanging errors on some pages.

This is using https://www.gaia-gis.it/gaia-sins/knn/tuscany_housenumbers.7z from the SpatiaLite KNN tutorial at https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
792384854 https://github.com/simonw/datasette/issues/1249#issuecomment-792384854 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDc5MjM4NDg1NA== simonw 9599 2021-03-08T00:23:38Z 2021-03-08T00:23:38Z OWNER

One reason to prioritize this issue: Homebrew upgraded to SpatiaLite 5.0 recently https://formulae.brew.sh/formula/spatialite-tools and as a result SpatiaLite database created on my laptop don't appear to be compatible with Datasette when published using datasette publish.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
792384382 https://github.com/simonw/datasette/issues/1249#issuecomment-792384382 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDc5MjM4NDM4Mg== simonw 9599 2021-03-08T00:22:02Z 2021-03-08T00:22:02Z OWNER

I tried this patch against Dockerfile:

diff --git a/Dockerfile b/Dockerfile
index f4b1414..dd659e1 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,25 +1,26 @@
-FROM python:3.7.10-slim-stretch as build
+FROM python:3.9.2-slim-buster as build

 # Setup build dependencies
 RUN apt update \
-&& apt install -y python3-dev build-essential wget libxml2-dev libproj-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
- && apt clean
+  && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
+  libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
+  && apt clean

-
-RUN wget "https://www.sqlite.org/2020/sqlite-autoconf-3310100.tar.gz" && tar xzf sqlite-autoconf-3310100.tar.gz \
-    && cd sqlite-autoconf-3310100 && ./configure --disable-static --enable-fts5 --enable-json1 CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1" \
+RUN wget "https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz" && tar xzf sqlite-autoconf-3340100.tar.gz \
+    && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
+    CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1" \
     && make && make install

-RUN wget "http://www.gaia-gis.it/gaia-sins/freexl-sources/freexl-1.0.5.tar.gz" && tar zxf freexl-1.0.5.tar.gz \
-    && cd freexl-1.0.5 && ./configure && make && make install
+RUN wget "http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz" && tar zxf freexl-1.0.6.tar.gz \
+    && cd freexl-1.0.6 && ./configure && make && make install

-RUN wget "http://www.gaia-gis.it/gaia-sins/libspatialite-sources/libspatialite-4.4.0-RC0.tar.gz" && tar zxf libspatialite-4.4.0-RC0.tar.gz \
-    && cd libspatialite-4.4.0-RC0 && ./configure && make && make install
+RUN wget "http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz" && tar zxf libspatialite-5.0.1.tar.gz \
+    && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install

 RUN wget "http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install

-RUN wget "http://www.gaia-gis.it/gaia-sins/spatialite-tools-sources/spatialite-tools-4.4.0-RC0.tar.gz" && tar zxf spatialite-tools-4.4.0-RC0.tar.gz \
-    && cd spatialite-tools-4.4.0-RC0 && ./configure && make && make install
+RUN wget "http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz" && tar zxf spatialite-tools-5.0.0.tar.gz \
+    && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install


 # Add local code to the image instead of fetching from pypi.
@@ -27,7 +28,7 @@ COPY . /datasette

 RUN pip install /datasette

-FROM python:3.7.10-slim-stretch
+FROM python:3.9.2-slim-buster

 # Copy python dependencies and spatialite libraries
 COPY --from=build /usr/local/lib/ /usr/local/lib/

I had to use --disable-rttopo from the tip in https://github.com/OSGeo/gdal/pull/3443 and also needed to install libminizip-dev.

This works, sort of... I'm getting a weird issue where the /dbname page is hanging some of the time instead of loading correctly. Other than that it seems to work, but a hanging page is bad!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
792383956 https://github.com/simonw/datasette/issues/1249#issuecomment-792383956 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDc5MjM4Mzk1Ng== simonw 9599 2021-03-08T00:20:09Z 2021-03-08T00:20:09Z OWNER

Worth noting that the Docker image used by datasette publish cloudrun doesn't actually use that Datasette docker image - it does this:

https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/__init__.py#L349-L353

Where the apt extras for SpatiaLite are: https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/__init__.py#L344-L345

libsqlite3-mod-spatialite against that official python:3.8 image doesn't appear to install SpatiaLite 5.0.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);