5,578 rows sorted by updated_at descending

View and edit SQL

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
808979218 https://github.com/simonw/datasette/issues/1276#issuecomment-808979218 https://api.github.com/repos/simonw/datasette/issues/1276 MDEyOklzc3VlQ29tbWVudDgwODk3OTIxOA== simonw 9599 2021-03-28T23:35:53Z 2021-03-28T23:36:26Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Invalid SQL: "no such table: pragma_database_list" on database page 841456306  
808979049 https://github.com/simonw/datasette/issues/1276#issuecomment-808979049 https://api.github.com/repos/simonw/datasette/issues/1276 MDEyOklzc3VlQ29tbWVudDgwODk3OTA0OQ== simonw 9599 2021-03-28T23:34:38Z 2021-03-28T23:34:38Z OWNER

The Glitch server logs showed:

ERROR: conn=<sqlite3.Connection object at 0x7f88ebadd8f0>, sql = 'select seq, name, file from pragma_database_list() where seq > 0', params = None: no such table: pragma_database_list

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Invalid SQL: "no such table: pragma_database_list" on database page 841456306  
808978808 https://github.com/simonw/datasette/issues/1276#issuecomment-808978808 https://api.github.com/repos/simonw/datasette/issues/1276 MDEyOklzc3VlQ29tbWVudDgwODk3ODgwOA== simonw 9599 2021-03-28T23:32:58Z 2021-03-28T23:33:58Z OWNER

I just managed to replicate this bug on Glitch: https://nicar-2020.glitch.me/data

Invalid SQL
no such table: pragma_database_list

https://nicar-2020.glitch.me/-/versions says:

{
    "python": {
        "version": "3.7.10",
        "full": "3.7.10 (default, Feb 20 2021, 21:21:24) \n[GCC 5.4.0 20160609]"
    },
    "datasette": {
        "version": "0.55"
    },
    "asgi": "3.0",
    "uvicorn": "0.13.4",
    "sqlite": {
        "version": "3.11.0",
        "fts_versions": [
            "FTS4",
            "FTS3"
        ],
        "extensions": {
            "json1": null
        },
        "compile_options": [
            "ENABLE_COLUMN_METADATA",
            "ENABLE_DBSTAT_VTAB",
            "ENABLE_FTS3",
            "ENABLE_FTS3_PARENTHESIS",
            "ENABLE_JSON1",
            "ENABLE_LOAD_EXTENSION",
            "ENABLE_RTREE",
            "ENABLE_UNLOCK_NOTIFY",
            "ENABLE_UPDATE_DELETE_LIMIT",
            "HAVE_ISNAN",
            "LIKE_DOESNT_MATCH_BLOBS",
            "MAX_SCHEMA_RETRY=25",
            "OMIT_LOOKASIDE",
            "SECURE_DELETE",
            "SOUNDEX",
            "SYSTEM_MALLOC",
            "TEMP_STORE=1",
            "THREADSAFE=1"
        ]
    }
}

That's SQLite 3.11.0 from 2016-02-15 with no FTS5.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Invalid SQL: "no such table: pragma_database_list" on database page 841456306  
808762613 https://github.com/simonw/datasette/pull/1279#issuecomment-808762613 https://api.github.com/repos/simonw/datasette/issues/1279 MDEyOklzc3VlQ29tbWVudDgwODc2MjYxMw== codecov[bot] 22429695 2021-03-27T17:03:37Z 2021-03-27T17:03:37Z NONE

Codecov Report

Merging #1279 (14d8977) into main (3fcfc85) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1279   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3fcfc85...14d8977. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Minor Docs Update. Added `--app` to fly install command. 842556944  
808759984 https://github.com/simonw/datasette/issues/1273#issuecomment-808759984 https://api.github.com/repos/simonw/datasette/issues/1273 MDEyOklzc3VlQ29tbWVudDgwODc1OTk4NA== simonw 9599 2021-03-27T16:43:17Z 2021-03-27T16:43:17Z OWNER

That rivers example in the tutorial would work a lot better with a live demo.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refresh SpatiaLite documentation 838382890  
808756921 https://github.com/simonw/datasette/issues/1273#issuecomment-808756921 https://api.github.com/repos/simonw/datasette/issues/1273 MDEyOklzc3VlQ29tbWVudDgwODc1NjkyMQ== simonw 9599 2021-03-27T16:19:45Z 2021-03-27T16:26:28Z OWNER

I have a better recipe for using spatial indexes now on https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/

select
  AsGeoJSON(geometry), *
from
  CPAD_2020a_SuperUnits
where
  PARK_NAME like '%mini%' and
  Intersects(GeomFromGeoJSON(:freedraw), geometry) = 1
  and CPAD_2020a_SuperUnits.rowid in (
    select
      rowid
    from
      SpatialIndex
    where
      f_table_name = 'CPAD_2020a_SuperUnits'
      and search_frame = GeomFromGeoJSON(:freedraw)
  )
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refresh SpatiaLite documentation 838382890  
808757721 https://github.com/simonw/datasette/issues/1273#issuecomment-808757721 https://api.github.com/repos/simonw/datasette/issues/1273 MDEyOklzc3VlQ29tbWVudDgwODc1NzcyMQ== simonw 9599 2021-03-27T16:25:48Z 2021-03-27T16:25:48Z OWNER

This will give you back an additional column of GeoJSON. You can copy and paste GeoJSON from this column into the debugging tool at geojson.io to visualize it on a map.

That should promote datasette-leaflet-geojson instead.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refresh SpatiaLite documentation 838382890  
808757659 https://github.com/simonw/datasette/issues/1090#issuecomment-808757659 https://api.github.com/repos/simonw/datasette/issues/1090 MDEyOklzc3VlQ29tbWVudDgwODc1NzY1OQ== simonw 9599 2021-03-27T16:25:25Z 2021-03-27T16:25:25Z OWNER

Related feature request: ability to set default values for canned queries: #1258

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Custom widgets for canned query forms 741862364  
808757155 https://github.com/simonw/datasette/issues/1090#issuecomment-808757155 https://api.github.com/repos/simonw/datasette/issues/1090 MDEyOklzc3VlQ29tbWVudDgwODc1NzE1NQ== simonw 9599 2021-03-27T16:21:43Z 2021-03-27T16:21:43Z OWNER

Idea for these: imitate https://django-sql-dashboard.readthedocs.io/en/latest/widgets.html#custom-widgets and drive them with templates.

So a custom widget type of textarea would look for a template called widgets/textarea.html - which means users could define brand new custom widgets just by creating their own template files.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Custom widgets for canned query forms 741862364  
808756366 https://github.com/simonw/datasette/issues/1278#issuecomment-808756366 https://api.github.com/repos/simonw/datasette/issues/1278 MDEyOklzc3VlQ29tbWVudDgwODc1NjM2Ng== simonw 9599 2021-03-27T16:15:47Z 2021-03-27T16:15:47Z OWNER

https://timezones-api.datasette.io/ is now up and running on Cloud Run.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
SpatiaLite timezones demo is broken 842416110  
808652008 https://github.com/simonw/datasette/issues/1278#issuecomment-808652008 https://api.github.com/repos/simonw/datasette/issues/1278 MDEyOklzc3VlQ29tbWVudDgwODY1MjAwOA== simonw 9599 2021-03-27T04:47:17Z 2021-03-27T04:47:17Z OWNER

https://github.com/simonw/timezones-api is that project, it's pretty old now. I'll try to get it running on Cloud Run.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
SpatiaLite timezones demo is broken 842416110  
808651088 https://github.com/simonw/datasette/issues/1258#issuecomment-808651088 https://api.github.com/repos/simonw/datasette/issues/1258 MDEyOklzc3VlQ29tbWVudDgwODY1MTA4OA== simonw 9599 2021-03-27T04:41:52Z 2021-03-27T04:42:14Z OWNER

Right now they look like this:

databases:
  fixtures:
    queries:
      neighborhood_search:
        params:
        - text

In addition to being able to specify defaults, I'd also like to add other things in the future - most significantly the ability to specify a different input widget (e.g. textarea v.s. single-line input)

So maybe this looks like:

params:
- name: text
  default: ""
- name: age
  widget: number
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow canned query params to specify default values 828858421  
808650266 https://github.com/simonw/datasette/issues/1258#issuecomment-808650266 https://api.github.com/repos/simonw/datasette/issues/1258 MDEyOklzc3VlQ29tbWVudDgwODY1MDI2Ng== simonw 9599 2021-03-27T04:37:07Z 2021-03-27T04:37:07Z OWNER

I like that idea.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow canned query params to specify default values 828858421  
808649480 https://github.com/simonw/datasette/issues/1249#issuecomment-808649480 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwODY0OTQ4MA== simonw 9599 2021-03-27T04:32:10Z 2021-03-27T04:32:10Z OWNER

I'll close this issue after I ship Datasette 0.56 and confirm that the Dockerfile was correctly built and published to Docker Hub.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
808649322 https://github.com/simonw/datasette/issues/1249#issuecomment-808649322 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwODY0OTMyMg== simonw 9599 2021-03-27T04:31:28Z 2021-03-27T04:31:28Z OWNER

One last test of that Dockerfile:

(datasette) datasette % docker build -f Dockerfile -t datasetteproject/datasette:0.55a --build-arg VERSION=0.55 .
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' | jq
{
  "python": {
    "version": "3.9.2",
    "full": "3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"
  },
  "datasette": {
    "version": "0.55"
  },
  "asgi": "3.0",
  "uvicorn": "0.13.4",
  "sqlite": {
    "version": "3.27.2",
    "fts_versions": [
      "FTS5",
      "FTS4",
      "FTS3"
    ],
    "extensions": {
      "json1": null
    },
    "compile_options": [
      "COMPILER=gcc-8.3.0",
      "ENABLE_COLUMN_METADATA",
      "ENABLE_DBSTAT_VTAB",
      "ENABLE_FTS3",
      "ENABLE_FTS3_PARENTHESIS",
      "ENABLE_FTS3_TOKENIZER",
      "ENABLE_FTS4",
      "ENABLE_FTS5",
      "ENABLE_JSON1",
      "ENABLE_LOAD_EXTENSION",
      "ENABLE_PREUPDATE_HOOK",
      "ENABLE_RTREE",
      "ENABLE_SESSION",
      "ENABLE_STMTVTAB",
      "ENABLE_UNLOCK_NOTIFY",
      "ENABLE_UPDATE_DELETE_LIMIT",
      "HAVE_ISNAN",
      "LIKE_DOESNT_MATCH_BLOBS",
      "MAX_SCHEMA_RETRY=25",
      "MAX_VARIABLE_NUMBER=250000",
      "OMIT_LOOKASIDE",
      "SECURE_DELETE",
      "SOUNDEX",
      "TEMP_STORE=1",
      "THREADSAFE=1",
      "USE_URI"
    ]
  }
}
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' --load-extension=spatialite | jq
{
  "python": {
    "version": "3.9.2",
    "full": "3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"
  },
  "datasette": {
    "version": "0.55"
  },
  "asgi": "3.0",
  "uvicorn": "0.13.4",
  "sqlite": {
    "version": "3.27.2",
    "fts_versions": [
      "FTS5",
      "FTS4",
      "FTS3"
    ],
    "extensions": {
      "json1": null,
      "spatialite": "5.0.1"
    },
    "compile_options": [
      "COMPILER=gcc-8.3.0",
      "ENABLE_COLUMN_METADATA",
      "ENABLE_DBSTAT_VTAB",
      "ENABLE_FTS3",
      "ENABLE_FTS3_PARENTHESIS",
      "ENABLE_FTS3_TOKENIZER",
      "ENABLE_FTS4",
      "ENABLE_FTS5",
      "ENABLE_JSON1",
      "ENABLE_LOAD_EXTENSION",
      "ENABLE_PREUPDATE_HOOK",
      "ENABLE_RTREE",
      "ENABLE_SESSION",
      "ENABLE_STMTVTAB",
      "ENABLE_UNLOCK_NOTIFY",
      "ENABLE_UPDATE_DELETE_LIMIT",
      "HAVE_ISNAN",
      "LIKE_DOESNT_MATCH_BLOBS",
      "MAX_SCHEMA_RETRY=25",
      "MAX_VARIABLE_NUMBER=250000",
      "OMIT_LOOKASIDE",
      "SECURE_DELETE",
      "SOUNDEX",
      "TEMP_STORE=1",
      "THREADSAFE=1",
      "USE_URI"
    ]
  }
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
808648974 https://github.com/simonw/datasette/issues/1272#issuecomment-808648974 https://api.github.com/repos/simonw/datasette/issues/1272 MDEyOklzc3VlQ29tbWVudDgwODY0ODk3NA== simonw 9599 2021-03-27T04:29:42Z 2021-03-27T04:29:42Z OWNER

I'm skipping this for the moment because the new Dockerfile shape introduced in https://github.com/simonw/datasette/issues/1249#issuecomment-804404544 isn't compatible with this technique, since it installs Datasette from PyPI rather than directly from the repo.

Will need to change that if I want to do this unit tests thing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Unit tests for the Dockerfile 838245338  
808647937 https://github.com/simonw/datasette/issues/1272#issuecomment-808647937 https://api.github.com/repos/simonw/datasette/issues/1272 MDEyOklzc3VlQ29tbWVudDgwODY0NzkzNw== simonw 9599 2021-03-27T04:23:19Z 2021-03-27T04:23:36Z OWNER

Part of the challenge here is only running if a Docker daemon is available. I think this pattern works, in tests/test_dockerfile.py:

import httpx
import pathlib
import pytest
import subprocess

root = pathlib.Path(__file__).parent.parent

def docker_is_available():
    try:
        client = httpx.Client(
            transport=httpx.HTTPTransport(uds="/var/run/docker.sock")
        )
        client.get("http://docker/info")
        return True
    except httpx.ConnectError:
        return False


@pytest.fixture
def build_container():
    assert (root / "Dockerfile").exists()
    subprocess.check_call([
        "docker", "build", str(root), "-t", "datasette-dockerfile-test"
    ])


@pytest.mark.skipif(not docker_is_available(),
    reason="Docker is not available"
)
def test_dockerfile(build_container):
    output = subprocess.check_output([
        "docker", "run", "datasette-dockerfile-test", "datasette", "--get", "/_memory?sql=select+1&shape=_array"
    ])
    assert False, "Implement better assertion here"
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Unit tests for the Dockerfile 838245338  
808642405 https://github.com/simonw/datasette/issues/1276#issuecomment-808642405 https://api.github.com/repos/simonw/datasette/issues/1276 MDEyOklzc3VlQ29tbWVudDgwODY0MjQwNQ== simonw 9599 2021-03-27T03:53:18Z 2021-03-27T03:53:18Z OWNER

That's really odd. What version of SQLite are you using on the server? You can tell by visiting https://your-site/-/versions

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Invalid SQL: "no such table: pragma_database_list" on database page 841456306  
808641846 https://github.com/simonw/datasette/issues/1277#issuecomment-808641846 https://api.github.com/repos/simonw/datasette/issues/1277 MDEyOklzc3VlQ29tbWVudDgwODY0MTg0Ng== simonw 9599 2021-03-27T03:49:34Z 2021-03-27T03:49:34Z OWNER

I fixed this already, it's a duplicate of #1239

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Facet by array breaks if table name contains a space 842212586  
808302971 https://github.com/simonw/sqlite-utils/issues/252#issuecomment-808302971 https://api.github.com/repos/simonw/sqlite-utils/issues/252 MDEyOklzc3VlQ29tbWVudDgwODMwMjk3MQ== simonw 9599 2021-03-26T15:21:38Z 2021-03-26T15:21:38Z OWNER

Already got that! It's the --nl option - works for both importing and exporting data: https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support json-line files 842062949  
807647791 https://github.com/simonw/sqlite-utils/issues/251#issuecomment-807647791 https://api.github.com/repos/simonw/sqlite-utils/issues/251 MDEyOklzc3VlQ29tbWVudDgwNzY0Nzc5MQ== simonw 9599 2021-03-25T22:42:48Z 2021-03-25T22:44:31Z OWNER

Idea: enhance lambda to allow it to return a dictionary of values, which will then be used to populate new columns. Use a --multicolumn option to indicate this:

sqlite-utils convert lambda mydb.db mytable mycolumn \
  --code '{"first_name": value.split()[0], "last_name": value.split()[1]}' \
  --multicolumn --drop

The --drop means "drop the mycolumn column after making this change".

Maybe --multi is a better name than --multicolumn here, since either way it's going to need additional explanation somewhere.

Would this overlap with #239 at all?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"sqlite-utils convert" command to replace the separate "sqlite-transform" tool 841377702  
807642041 https://github.com/simonw/sqlite-utils/issues/251#issuecomment-807642041 https://api.github.com/repos/simonw/sqlite-utils/issues/251 MDEyOklzc3VlQ29tbWVudDgwNzY0MjA0MQ== simonw 9599 2021-03-25T22:39:22Z 2021-03-25T22:39:22Z OWNER

Here's the full current implementation of that tool: https://github.com/simonw/sqlite-transform/blob/0.5/sqlite_transform/cli.py

My current plan is to make this functionality available as the following:

sqlite-utils convert jsonsplit mydb.db mytable mycolumn
sqlite-utils convert parsedatetime mydb.db mytable mycolumn
sqlite-utils convert parsedate mydb.db mytable mycolumn
sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()'
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"sqlite-utils convert" command to replace the separate "sqlite-transform" tool 841377702  
807459633 https://github.com/simonw/datasette/issues/1258#issuecomment-807459633 https://api.github.com/repos/simonw/datasette/issues/1258 MDEyOklzc3VlQ29tbWVudDgwNzQ1OTYzMw== wdccdw 1385831 2021-03-25T20:48:33Z 2021-03-25T20:49:34Z NONE

What about allowing default parameters when defining the query in metadata.yml? Something like:

databases:
  fec:
    queries:
      search_by_name:
        params:
            - q
        default-param-values:
             q: "text to search"
        sql: |-
          SELECT...

For now, I'm using a custom database-<file>.html file that hardcodes a default param in the link, but I'd rather not customize the template just for that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow canned query params to specify default values 828858421  
806166575 https://github.com/simonw/datasette/issues/741#issuecomment-806166575 https://api.github.com/repos/simonw/datasette/issues/741 MDEyOklzc3VlQ29tbWVudDgwNjE2NjU3NQ== simonw 9599 2021-03-24T20:30:33Z 2021-03-24T20:30:33Z OWNER

datasette package is a mostly unmaintained feature at this point - it has a bit of test coverage but I've not made any improvements to it in a few years, and I don't use it for my own projects.

I'll make this change to package at the same time as I land it for publish though.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Replace "datasette publish --extra-options" with "--setting" 607223136  
806010960 https://github.com/simonw/datasette/issues/741#issuecomment-806010960 https://api.github.com/repos/simonw/datasette/issues/741 MDEyOklzc3VlQ29tbWVudDgwNjAxMDk2MA== zaneselvans 596279 2021-03-24T17:19:42Z 2021-03-24T17:19:42Z NONE

Ah, okay so --extra-options applies to both datasette publish and datasette package? There wren't any examples of it being used with publish in the docs, so this tripped me up for a bit.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Replace "datasette publish --extra-options" with "--setting" 607223136  
805216038 https://github.com/simonw/datasette/issues/1274#issuecomment-805216038 https://api.github.com/repos/simonw/datasette/issues/1274 MDEyOklzc3VlQ29tbWVudDgwNTIxNjAzOA== simonw 9599 2021-03-23T20:14:53Z 2021-03-23T20:14:53Z OWNER

Yes this is one of the main reasons I'm planning to switch to encouraging YAML be default instead of JSON (while still supporting JSON) - YAML supports comments and multi-line strings.

See #1153 for YAML by default in the documentation.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Might there be some way to comment metadata.json? 839008371  
805214307 https://github.com/simonw/datasette/issues/1274#issuecomment-805214307 https://api.github.com/repos/simonw/datasette/issues/1274 MDEyOklzc3VlQ29tbWVudDgwNTIxNDMwNw== bobwhitelock 7476523 2021-03-23T20:12:29Z 2021-03-23T20:12:29Z CONTRIBUTOR

One issue I could see with adding first class support for metadata in hjson format is that this would require adding an additional dependency to handle this, for a feature that would be unused by many users. I wonder if this could fit in as a plugin instead; if a hook existed for loading metadata (maybe as part of https://github.com/simonw/datasette/issues/860) the metadata could then come from any source, as specified by plugins, e.g. hjson, toml, XML, a database table etc.

Until/unless this exists, a few ideas for how you could add comments:
- Using YAML as you suggest.
- A common pattern is adding a "comment" key for comments to any object in JSON - I don't think including an unnecessary key like this would break anything in Datasette, but not certain.
- You could use another tool as a preprocessor for your JSON metadata - e.g. hjson or Jsonnet. You'd write the metadata in that format, and then convert that into JSON to actually use as your final metadata.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Might there be some way to comment metadata.json? 839008371  
805109341 https://github.com/simonw/datasette/issues/1153#issuecomment-805109341 https://api.github.com/repos/simonw/datasette/issues/1153 MDEyOklzc3VlQ29tbWVudDgwNTEwOTM0MQ== simonw 9599 2021-03-23T17:55:48Z 2021-03-23T18:41:57Z OWNER

Beginnings of a UI element for switching between them:

<div style="border: 1px solid rgb(225, 228, 229);
  background-color: rgb(238, 255, 204);
  padding: 0.3em;
  position: relative;
  top: 3px;
  font-family: courier;">
<a href="#" style="display: inline-block; padding-left: 0px; padding-right: 2em;">JSON</a>
<a href="#" style="display: inline-block;">YAML</a>
</div>

https://user-images.githubusercontent.com/9599/112194637-51f92500-8bc6-11eb-9662-3faa7ef37538.png">

That <pre> has a padding of 12px, so using 12px padding on the tab links should get them to line up better.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use YAML examples in documentation by default, not JSON 771202454  
805047117 https://github.com/simonw/datasette/issues/1153#issuecomment-805047117 https://api.github.com/repos/simonw/datasette/issues/1153 MDEyOklzc3VlQ29tbWVudDgwNTA0NzExNw== simonw 9599 2021-03-23T16:30:15Z 2021-03-23T16:46:06Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use YAML examples in documentation by default, not JSON 771202454  
805058241 https://github.com/simonw/datasette/issues/1270#issuecomment-805058241 https://api.github.com/repos/simonw/datasette/issues/1270 MDEyOklzc3VlQ29tbWVudDgwNTA1ODI0MQ== simonw 9599 2021-03-23T16:45:39Z 2021-03-23T16:45:39Z OWNER

I managed to build SpatiaLite such that this isn't necessary any more. I'm still interested in pursuing this further though - it feels like it could be a more robust way of implementing timeouts, but I need to prove to myself that it's better (maybe better performance, or handles more edge-cases?). Not sure how to prove that yet.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler() 837350092  
805056806 https://github.com/simonw/datasette/issues/1153#issuecomment-805056806 https://api.github.com/repos/simonw/datasette/issues/1153 MDEyOklzc3VlQ29tbWVudDgwNTA1NjgwNg== simonw 9599 2021-03-23T16:43:38Z 2021-03-23T16:43:38Z OWNER

I used this code to get that:

var jsonVersion = JSON.stringify(window.jsyaml.load(document.querySelector('.highlight-yaml').textContent), null, 4);
div.querySelector('.highlight pre').innerText = jsonVersion;
div.querySelector('.highlight pre').style.whiteSpace = 'pre-wrap'
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use YAML examples in documentation by default, not JSON 771202454  
805055291 https://github.com/simonw/datasette/issues/1153#issuecomment-805055291 https://api.github.com/repos/simonw/datasette/issues/1153 MDEyOklzc3VlQ29tbWVudDgwNTA1NTI5MQ== simonw 9599 2021-03-23T16:41:31Z 2021-03-23T16:41:31Z OWNER

One downside of doing this conversion in JavaScript: it's much harder to get the same JSON syntax highlighting as that provided by Sphinx:

https://user-images.githubusercontent.com/9599/112183909-ef9b2700-8bbb-11eb-882b-aeb387c29e61.png">

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use YAML examples in documentation by default, not JSON 771202454  
805050163 https://github.com/simonw/datasette/issues/1153#issuecomment-805050163 https://api.github.com/repos/simonw/datasette/issues/1153 MDEyOklzc3VlQ29tbWVudDgwNTA1MDE2Mw== simonw 9599 2021-03-23T16:34:35Z 2021-03-23T16:35:32Z OWNER

https://docs.datasette.io/en/stable/metadata.html has this example:

title: Demonstrating Metadata from YAML
description_html: |-
  <p>This description includes a long HTML string</p>
  <ul>
    <li>YAML is better for embedding HTML strings than JSON!</li>
  </ul>
license: ODbL
license_url: https://opendatacommons.org/licenses/odbl/
databases:
  fixtures:
    tables:
      no_primary_key:
        hidden: true
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%' order by neighborhood;
        title: Search neighborhoods
        description_html: |-
          <p>This demonstrates <em>basic</em> LIKE search

I ran this in the browser dev tools:

var s = document.createElement('script')
s.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js'
document.head.appendChild(s)
var yamlExample = document.querySelector('.highlight-yaml').textContent);
console.log(JSON.stringify(window.jsyaml.load(yamlExample), null, 4))

And got:

{
    "title": "Demonstrating Metadata from YAML",
    "description_html": "<p>This description includes a long HTML string</p>\n<ul>\n  <li>YAML is better for embedding HTML strings than JSON!</li>\n</ul>",
    "license": "ODbL",
    "license_url": "https://opendatacommons.org/licenses/odbl/",
    "databases": {
        "fixtures": {
            "tables": {
                "no_primary_key": {
                    "hidden": true
                }
            },
            "queries": {
                "neighborhood_search": {
                    "sql": "select neighborhood, facet_cities.name, state\nfrom facetable join facet_cities on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%' order by neighborhood;",
                    "title": "Search neighborhoods",
                    "description_html": "<p>This demonstrates <em>basic</em> LIKE search"
                }
            }
        }
    }
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use YAML examples in documentation by default, not JSON 771202454  
805042880 https://github.com/simonw/datasette/issues/1153#issuecomment-805042880 https://api.github.com/repos/simonw/datasette/issues/1153 MDEyOklzc3VlQ29tbWVudDgwNTA0Mjg4MA== simonw 9599 2021-03-23T16:24:32Z 2021-03-23T16:24:32Z OWNER

... actually I think I would do that conversion in Python. The client-side YAML parsers all look a little bit heavy to me in terms of additional page weight.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use YAML examples in documentation by default, not JSON 771202454  
805041522 https://github.com/simonw/datasette/issues/1153#issuecomment-805041522 https://api.github.com/repos/simonw/datasette/issues/1153 MDEyOklzc3VlQ29tbWVudDgwNTA0MTUyMg== simonw 9599 2021-03-23T16:22:46Z 2021-03-23T16:22:46Z OWNER

That's a good idea. I could do that with JavaScript - loading YAML and converting it to JSON in JavaScript shouldn't be hard, and it's better than JSON-to-YAML because there's only one correct JSON representation of a YAML file whereas you can represent a JSON document in YAML in a bunch of different ways.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use YAML examples in documentation by default, not JSON 771202454  
805033155 https://github.com/simonw/datasette/issues/1249#issuecomment-805033155 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNTAzMzE1NQ== simonw 9599 2021-03-23T16:12:13Z 2021-03-23T16:12:13Z OWNER

Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source

The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful VirtualKNN module), take a look at the Datasette Dockerfile.

See also #1273

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804698315 https://github.com/simonw/datasette/pull/1159#issuecomment-804698315 https://api.github.com/repos/simonw/datasette/issues/1159 MDEyOklzc3VlQ29tbWVudDgwNDY5ODMxNQ== lovasoa 552629 2021-03-23T07:58:28Z 2021-03-23T07:58:38Z NONE

@mroswell Did you try it with more columns ? The display is flexible and columns get closer as new ones are added.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improve the display of facets information 774332247  
804640440 https://github.com/simonw/datasette/issues/1153#issuecomment-804640440 https://api.github.com/repos/simonw/datasette/issues/1153 MDEyOklzc3VlQ29tbWVudDgwNDY0MDQ0MA== mroswell 192568 2021-03-23T05:58:20Z 2021-03-23T05:58:20Z CONTRIBUTOR

Could there be a little widget that offers conversion from one to the other?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use YAML examples in documentation by default, not JSON 771202454  
804639427 https://github.com/simonw/datasette/pull/1159#issuecomment-804639427 https://api.github.com/repos/simonw/datasette/issues/1159 MDEyOklzc3VlQ29tbWVudDgwNDYzOTQyNw== mroswell 192568 2021-03-23T05:56:02Z 2021-03-23T05:56:02Z CONTRIBUTOR

With just three facets, I like it, but it does take more horizontal space. Would be nice to have a switch somewhere, enabling either original compact option or this proposed more-readable option. Also some control over word wrap (width setting) and facet spacing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improve the display of facets information 774332247  
804541064 https://github.com/simonw/datasette/issues/164#issuecomment-804541064 https://api.github.com/repos/simonw/datasette/issues/164 MDEyOklzc3VlQ29tbWVudDgwNDU0MTA2NA== mroswell 192568 2021-03-23T02:45:12Z 2021-03-23T02:45:12Z CONTRIBUTOR

"datasette skeleton" feature removed #476

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette skeleton command for kick-starting database and table metadata 280013907  
804540869 https://github.com/simonw/datasette/issues/163#issuecomment-804540869 https://api.github.com/repos/simonw/datasette/issues/163 MDEyOklzc3VlQ29tbWVudDgwNDU0MDg2OQ== simonw 9599 2021-03-23T02:44:33Z 2021-03-23T02:44:33Z OWNER

Comments welcome!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Document the querystring argument for setting a different time limit 279547886  
804539729 https://github.com/simonw/datasette/issues/163#issuecomment-804539729 https://api.github.com/repos/simonw/datasette/issues/163 MDEyOklzc3VlQ29tbWVudDgwNDUzOTcyOQ== mroswell 192568 2021-03-23T02:41:14Z 2021-03-23T02:41:14Z CONTRIBUTOR

I'm visiting old issues for context while learning datasette. Let me know if okay to make the occasional comment like this one.
querystring argument now located at:
https://docs.datasette.io/en/latest/settings.html#sql-time-limit-ms

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Document the querystring argument for setting a different time limit 279547886  
804471733 https://github.com/simonw/datasette/issues/88#issuecomment-804471733 https://api.github.com/repos/simonw/datasette/issues/88 MDEyOklzc3VlQ29tbWVudDgwNDQ3MTczMw== mroswell 192568 2021-03-22T23:46:36Z 2021-03-22T23:46:36Z CONTRIBUTOR

Google Map API limits seem to prevent https://nhs-england-map.netlify.com from being a working demo.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add NHS England Hospitals example to wiki 273775212  
804299406 https://github.com/simonw/datasette/pull/1271#issuecomment-804299406 https://api.github.com/repos/simonw/datasette/issues/1271 MDEyOklzc3VlQ29tbWVudDgwNDI5OTQwNg== simonw 9599 2021-03-22T18:36:14Z 2021-03-22T21:49:27Z OWNER

This isn't actually working - the outer code attempts to send an .interrupt() call to the connection object via the connections thread-local, which doesn't work because it's a thread-local so the connection isn't visible to that code.

Need to figure out how to communicate with that thread properly.

Also a test that fails in this particular case would be a good idea!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use SQLite conn.interrupt() instead of sqlite_timelimit() 837956424  
804415619 https://github.com/simonw/datasette/issues/1149#issuecomment-804415619 https://api.github.com/repos/simonw/datasette/issues/1149 MDEyOklzc3VlQ29tbWVudDgwNDQxNTYxOQ== mroswell 192568 2021-03-22T21:43:16Z 2021-03-22T21:43:16Z CONTRIBUTOR

Sounds like a good idea.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make it easier to theme Datasette with CSS 769520939  
804406675 https://github.com/simonw/datasette/issues/1249#issuecomment-804406675 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDQwNjY3NQ== simonw 9599 2021-03-22T21:26:27Z 2021-03-22T21:26:27Z OWNER

(Without the apt-get update ... SpatiaLite line it's 125MB)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804404544 https://github.com/simonw/datasette/issues/1249#issuecomment-804404544 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA== simonw 9599 2021-03-22T21:22:56Z 2021-03-22T21:24:24Z OWNER

Final version of Dockerfile which installs the specified version from GitHub:

docker build . -t datasette-spatialite --build-arg VERSION=0.55
FROM python:3.9.2-slim-buster as build

# Version of Datasette to install, e.g. 0.55
#   docker build . -t datasette --build-arg VERSION=0.55
ARG VERSION

# software-properties-common provides add-apt-repository
# which we need in order to install a more recent release
# of libsqlite3-mod-spatialite from the sid distribution
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt-get remove -y software-properties-common && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD ["datasette"]

Run against 0.55 this produces an image of 262MB

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804384196 https://github.com/simonw/datasette/issues/1249#issuecomment-804384196 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng== simonw 9599 2021-03-22T20:48:46Z 2021-03-22T20:48:46Z OWNER

I think part of the reason it's smaller is that I ran pip install datasette instead of using COPY . /datasette followed by pip install /datasette.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804380181 https://github.com/simonw/datasette/issues/1249#issuecomment-804380181 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ== simonw 9599 2021-03-22T20:42:16Z 2021-03-22T20:42:16Z OWNER

Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804379644 https://github.com/simonw/datasette/issues/1249#issuecomment-804379644 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA== simonw 9599 2021-03-22T20:41:23Z 2021-03-22T20:41:23Z OWNER

I tried adding apt-get remove -y software-properties-common && to remove software-properties-common but it made no difference to the image size.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804372977 https://github.com/simonw/datasette/issues/1249#issuecomment-804372977 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw== simonw 9599 2021-03-22T20:30:37Z 2021-03-22T20:30:37Z OWNER

I tried copying just the mod_spatialite.so file into a second stage build but it failed. So I ran bash in a working image and used ldd to figure out what it was linked to:

root@39683f91e588:/usr/lib/x86_64-linux-gnu# ldd mod_spatialite.so
    linux-vdso.so.1 (0x00007ffd021f4000)
    libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f5c75412000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c753f0000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c752ac000)
    libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f5c750a0000)
    librttopo.so.1 => /usr/lib/x86_64-linux-gnu/librttopo.so.1 (0x00007f5c75028000)
    libfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f5c7501c000)
    libproj.so.19 => /usr/lib/x86_64-linux-gnu/libproj.so.19 (0x00007f5c74ca7000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c74a89000)
    libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f5c74967000)
    libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f5c7492b000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c74766000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c74760000)
    libicuuc.so.67 => /usr/lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007f5c74575000)
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5c7454d000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f5c75d49000)
    libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f5c744c7000)
    libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f5c74439000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5c7426c000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c74250000)
    libgeos-3.9.0.so => /usr/lib/x86_64-linux-gnu/libgeos-3.9.0.so (0x00007f5c74040000)
    libicudata.so.67 => /usr/lib/x86_64-linux-gnu/libicudata.so.67 (0x00007f5c72527000)
    libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f5c724bc000)
    libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f5c7241c000)
    libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f5c7220e000)
    libjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f5c72188000)
    libdeflate.so.0 => /usr/lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f5c7216c000)
    libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f5c72144000)
    libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f5c72125000)
    librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f5c71f08000)
    libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f5c71eda000)
    libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f5c71ec5000)
    libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f5c71e8d000)
    libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f5c71ce0000)
    libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f5c71c93000)
    libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f5c71bb3000)
    libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f5c71b7f000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f5c71b77000)
    libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f5c71b23000)
    liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f5c71b12000)
    libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f5c7198e000)
    libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f5c71955000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f5c718d0000)
    libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f5c717b2000)
    libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f5c71683000)
    libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f5c71470000)
    libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f5c71461000)
    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f5c71458000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5c7143e000)
    libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f5c71421000)
    libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5c713fe000)
    libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f5c713f4000)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804368372 https://github.com/simonw/datasette/issues/1249#issuecomment-804368372 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg== simonw 9599 2021-03-22T20:22:43Z 2021-03-22T20:22:43Z OWNER
FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD ["datasette"]

262 MB

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804363687 https://github.com/simonw/datasette/issues/1249#issuecomment-804363687 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw== simonw 9599 2021-03-22T20:15:00Z 2021-03-22T20:15:00Z OWNER
RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r

That dropped it to 265MB.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804360701 https://github.com/simonw/datasette/issues/1249#issuecomment-804360701 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM2MDcwMQ== simonw 9599 2021-03-22T20:10:07Z 2021-03-22T20:10:07Z OWNER

Adding --no-install-recommends dropped it to 275MB

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804347152 https://github.com/simonw/datasette/issues/1249#issuecomment-804347152 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg== simonw 9599 2021-03-22T19:47:56Z 2021-03-22T19:48:03Z OWNER

I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804344553 https://github.com/simonw/datasette/issues/1249#issuecomment-804344553 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDM0NDU1Mw== simonw 9599 2021-03-22T19:43:25Z 2021-03-22T19:43:25Z OWNER

Does --no-install-recommends make a difference?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804338678 https://github.com/simonw/datasette/issues/1249#issuecomment-804338678 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA== simonw 9599 2021-03-22T19:33:43Z 2021-03-22T19:33:43Z OWNER

Replacing rm -rf /var/lib/{apt,dpkg,cache,log}/ with

    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

Got the size down to 305MB.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804318314 https://github.com/simonw/datasette/issues/1249#issuecomment-804318314 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDMxODMxNA== simonw 9599 2021-03-22T19:04:30Z 2021-03-22T19:04:30Z OWNER

Considering the image on Docker Hub right now is 383MB this is actually an improvement.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804317545 https://github.com/simonw/datasette/issues/1249#issuecomment-804317545 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ== simonw 9599 2021-03-22T19:03:22Z 2021-03-22T19:03:22Z OWNER

This Dockerfile:

FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y install software-properties-common && \
    add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
    apt-get update && \
    apt-get -t sid install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

EXPOSE 8001
CMD ["datasette"]

Produces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804310353 https://github.com/simonw/datasette/issues/1249#issuecomment-804310353 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw== simonw 9599 2021-03-22T18:52:12Z 2021-03-22T18:52:12Z OWNER

This Dockerfile:

FROM python:3.9.2-slim-buster as build

# Setup build dependencies
RUN apt update \
    && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
    libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
    && apt clean

RUN wget "https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz" && tar xzf sqlite-autoconf-3340100.tar.gz \
    && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
    CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1" \
    && make && make install

RUN wget "http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz" && tar zxf freexl-1.0.6.tar.gz \
    && cd freexl-1.0.6 && ./configure && make && make install

RUN wget "http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz" && tar zxf libspatialite-5.0.1.tar.gz \
    && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install

RUN wget "http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install

RUN wget "http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz" && tar zxf spatialite-tools-5.0.0.tar.gz \
    && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install

# Add local code to the image instead of fetching from pypi.
#COPY . /datasette
#RUN pip install /datasette
RUN pip install datasette

FROM python:3.9.2-slim-buster

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD ["datasette"]

Produced a 448MB image.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804309510 https://github.com/simonw/datasette/issues/1249#issuecomment-804309510 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA== simonw 9599 2021-03-22T18:50:50Z 2021-03-22T18:50:50Z OWNER

Ideally I'd like to use the Debian stable python:3.9.2-slim-buster base image but install SpatiaLite from Debian unstable here: https://packages.debian.org/sid/libspatialite7

This pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804265042 https://github.com/simonw/datasette/pull/1271#issuecomment-804265042 https://api.github.com/repos/simonw/datasette/issues/1271 MDEyOklzc3VlQ29tbWVudDgwNDI2NTA0Mg== simonw 9599 2021-03-22T17:45:45Z 2021-03-22T17:45:45Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use SQLite conn.interrupt() instead of sqlite_timelimit() 837956424  
804263434 https://github.com/simonw/datasette/issues/1249#issuecomment-804263434 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA== simonw 9599 2021-03-22T17:43:25Z 2021-03-22T17:43:25Z OWNER

I figured out the cause of the hang in #1268 - it was caused by select count(*) from SpatialIndex interacting badly with the set_progress_handler() mechanism I was using to implement query time limits. #1271 has a replacement for that using asyncio.wait_for() and conn.interrupt() which should resolve the SpatiaLite issue too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
804261915 https://github.com/simonw/datasette/issues/1268#issuecomment-804261915 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwNDI2MTkxNQ== simonw 9599 2021-03-22T17:41:12Z 2021-03-22T17:41:12Z OWNER

Closing this because I've figured out the root of the problem now, and I have a potential solution.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
804261610 https://github.com/simonw/datasette/issues/1269#issuecomment-804261610 https://api.github.com/repos/simonw/datasette/issues/1269 MDEyOklzc3VlQ29tbWVudDgwNDI2MTYxMA== simonw 9599 2021-03-22T17:40:41Z 2021-03-22T17:40:41Z OWNER

1270 looks promising, and I don't want to leave open a security hole where someone could potentially hang Datasette with a nasty count(*) query.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't attempt to run count(*) against virtual tables 837348479  
804261103 https://github.com/simonw/datasette/pull/1271#issuecomment-804261103 https://api.github.com/repos/simonw/datasette/issues/1271 MDEyOklzc3VlQ29tbWVudDgwNDI2MTEwMw== codecov[bot] 22429695 2021-03-22T17:39:57Z 2021-03-22T17:39:57Z NONE

Codecov Report

Merging #1271 (fb2ad7a) into main (c4f1ec7) will decrease coverage by 0.28%.
The diff coverage is 94.28%.

@@            Coverage Diff             @@
##             main    #1271      +/-   ##
==========================================
- Coverage   91.51%   91.22%   -0.29%     
==========================================
  Files          34       34              
  Lines        4255     4263       +8     
==========================================
- Hits         3894     3889       -5     
- Misses        361      374      +13     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/database.py</td> <td>92.41% <94.28%> (-0.52%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/utils/__init__.py</td> <td>92.24% <0.00%> (-1.90%)</td> <td>:arrow_down:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4f1ec7...fb2ad7a. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use SQLite conn.interrupt() instead of sqlite_timelimit() 837956424  
804255633 https://github.com/simonw/datasette/issues/1270#issuecomment-804255633 https://api.github.com/repos/simonw/datasette/issues/1270 MDEyOklzc3VlQ29tbWVudDgwNDI1NTYzMw== simonw 9599 2021-03-22T17:32:02Z 2021-03-22T17:32:08Z OWNER

Confirmed that the interrupt() based cancellation mechanism fixes the SpatiaLite issue in #1268!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler() 837350092  
803834784 https://github.com/simonw/datasette/issues/1270#issuecomment-803834784 https://api.github.com/repos/simonw/datasette/issues/1270 MDEyOklzc3VlQ29tbWVudDgwMzgzNDc4NA== simonw 9599 2021-03-22T07:31:57Z 2021-03-22T16:22:19Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler() 837350092  
803802957 https://github.com/simonw/datasette/issues/1268#issuecomment-803802957 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzgwMjk1Nw== simonw 9599 2021-03-22T06:38:14Z 2021-03-22T06:38:14Z OWNER

Also worth trying is to change this code:

     n = 1000 
     if ms < 50: 
         n = 1 

What happens with n = 10 instead?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803785808 https://github.com/simonw/datasette/issues/1269#issuecomment-803785808 https://api.github.com/repos/simonw/datasette/issues/1269 MDEyOklzc3VlQ29tbWVudDgwMzc4NTgwOA== simonw 9599 2021-03-22T06:00:53Z 2021-03-22T06:00:53Z OWNER

This may not be necessary if using `.interrupt() for SQLite timeouts in #1270 works.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't attempt to run count(*) against virtual tables 837348479  
803784902 https://github.com/simonw/datasette/issues/1268#issuecomment-803784902 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc4NDkwMg== simonw 9599 2021-03-22T05:59:06Z 2021-03-22T05:59:06Z OWNER

Even if I implement that workaround in #1269 I'm concerned that this could still allow users to deliberately crash Datasette (if it's running SpatiaLite 5.0) by executing select count(*) from SpatialIndex.

That interrupt timeout mechanism is worth digging into further.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803782705 https://github.com/simonw/datasette/issues/1268#issuecomment-803782705 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc4MjcwNQ== simonw 9599 2021-03-22T05:54:19Z 2021-03-22T05:54:19Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803777724 https://github.com/simonw/datasette/issues/1268#issuecomment-803777724 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc3NzcyNA== simonw 9599 2021-03-22T05:42:50Z 2021-03-22T05:43:23Z OWNER

https://user-images.githubusercontent.com/9599/111945249-99809380-8a96-11eb-871b-9a1509360bcf.png">

If I want to avoid counting virtual tables, I need to detect which tables are virtual tables.

The safest way to do this is probably to pull the sql for every table and then, in Python, check for values that start with create virtual table after converting to lower case, using any number of spaces.

This would catch things like CREATE virtual TABLE which might be missed by a SQL like query.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803775121 https://github.com/simonw/datasette/issues/1268#issuecomment-803775121 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc3NTEyMQ== simonw 9599 2021-03-22T05:36:26Z 2021-03-22T05:36:26Z OWNER

So one fix could be to avoid running counts for anything that turns out to be a virtual table.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803774926 https://github.com/simonw/datasette/issues/1268#issuecomment-803774926 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc3NDkyNg== simonw 9599 2021-03-22T05:35:56Z 2021-03-22T05:35:56Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803774518 https://github.com/simonw/datasette/issues/1268#issuecomment-803774518 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc3NDUxOA== simonw 9599 2021-03-22T05:34:57Z 2021-03-22T05:34:57Z OWNER

... and sure enough, adding this code fixed the problem:

diff --git a/datasette/database.py b/datasette/database.py
index 3579cce..b466b12 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -224,6 +226,9 @@ class Database:
         # Try to get counts for each table, $limit timeout for each count
         counts = {}
         for table in await self.table_names():
+            if table == "SpatialIndex":
+                counts[table] = 0
+                continue
             try:
                 table_count = (
                     await self.execute(
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803773484 https://github.com/simonw/datasette/issues/1268#issuecomment-803773484 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc3MzQ4NA== simonw 9599 2021-03-22T05:32:29Z 2021-03-22T05:32:29Z OWNER

To figure out which SQL query triggers the problem I added this code to write to a log file:

            with sqlite_timelimit(conn, time_limit_ms):
                try:
                    cursor = conn.cursor()
                    with open("/tmp/sql.log", "ab", buffering=0) as fp:
                        fp.write(("{}: {}\n".format(sql, params)).encode("utf-8"))
                    cursor.execute(sql, params if params is not None else {})

I had to use ab binary mode because Python doesn't allow buffering=0 for non-binary file operations.

With the log enabled, I used docker exec -it 589ae68de943 bash to attach to the running container and tail -f /tmp/sql.log to see the logs. Here's where it broke:

select count(*) from [idx_civici_geom_parent]: None
select count(*) from [sqlite_stat1]: None
select count(*) from [sqlite_stat3]: None
select count(*) from [SpatialIndex]: None

So attempting to run a count(*) against the SpatialIndex virtual table is the thing that triggers the bug.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803764919 https://github.com/simonw/datasette/issues/1268#issuecomment-803764919 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc2NDkxOQ== simonw 9599 2021-03-22T05:11:11Z 2021-03-22T05:11:11Z OWNER

Maybe I could implement SQLite query timeouts using the interrupt() method instead of the progress handler hack I'm currently using?

https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803764200 https://github.com/simonw/datasette/issues/1268#issuecomment-803764200 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc2NDIwMA== simonw 9599 2021-03-22T05:09:13Z 2021-03-22T05:09:13Z OWNER

I tried building a container where the conn.set_progress_handler(handler, n) line was commented out... and it fixed the bug.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803762969 https://github.com/simonw/datasette/issues/1268#issuecomment-803762969 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc2Mjk2OQ== simonw 9599 2021-03-22T05:05:51Z 2021-03-22T05:05:51Z OWNER

I had to run docker kill 16197781a7b5 to kill the broken container - Ctrl+C in the Datasette console window didn't do anything.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803762609 https://github.com/simonw/datasette/issues/1268#issuecomment-803762609 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc2MjYwOQ== simonw 9599 2021-03-22T05:05:00Z 2021-03-22T05:05:00Z OWNER

Using https://til.simonwillison.net/docker/attach-bash-to-running-container - I figured out how to run gdb. I had to use --privileged here because otherwise gdb showed a "Could not attach to process" error.

docker exec --privileged -it 16197781a7b5 bash
# apt-get install gdb python3-dbg
# gdb /usr/bin/python3 -p 20

This paused the process. I tried running this:

(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1845, in _run_once
    if handle._cancelled:
  File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/server.py", line 49, in run
    loop.run_until_complete(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/main.py", line 386, in run
    server.run()
  File "/usr/local/lib/python3.8/dist-packages/datasette/cli.py", line 575, in serve
    uvicorn.run(ds.app(), **uvicorn_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/bin/datasette", line 8, in <module>
    sys.exit(cli())
  <built-in method exec of module object at remote 0x7f0981a280e0>
  File "/usr/lib/python3.8/trace.py", line 450, in runctx
    exec(cmd, globals, locals)
  File "/usr/lib/python3.8/trace.py", line 6632, in main
  File "/usr/lib/python3.8/trace.py", line 756, in <module>
    main()
  <built-in method exec of module object at remote 0x7f0981a280e0>
  File "/usr/lib/python3.8/runpy.py", line 343, in _run_code
  File "/usr/lib/python3.8/runpy.py", line 450, in _run_module_as_main

Not sure if that's useful or not.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803759051 https://github.com/simonw/datasette/issues/1268#issuecomment-803759051 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc1OTA1MQ== simonw 9599 2021-03-22T04:55:22Z 2021-03-22T04:55:22Z OWNER

So I think there's a bug in the way the set_progress_handler() mechanism works when used in conjunction with SpatiaLite 5.0 on Linux.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803758793 https://github.com/simonw/datasette/issues/1268#issuecomment-803758793 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc1ODc5Mw== simonw 9599 2021-03-22T04:54:32Z 2021-03-22T04:54:32Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803758182 https://github.com/simonw/datasette/issues/1268#issuecomment-803758182 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc1ODE4Mg== simonw 9599 2021-03-22T04:52:15Z 2021-03-22T04:52:15Z OWNER

Hitting http://localhost:8001/ successfully shows the homepage (after a lot more scrolling).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803757746 https://github.com/simonw/datasette/issues/1268#issuecomment-803757746 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc1Nzc0Ng== simonw 9599 2021-03-22T04:50:40Z 2021-03-22T04:51:52Z OWNER

Here's a fun debugging trick:

docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
root@16197781a7b5:/# python3 -m trace --trace $(which datasette) \
  -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite \
  --load-extension=spatialite

A huge amount of stuff scrolls past as Datasette starts up, since we are tracing every executed line of Python.

After about a minute it's finished starting and gets to this point:

selectors.py(452):             if timeout is None:
selectors.py(454):             elif timeout <= 0:
selectors.py(459):                 timeout = math.ceil(timeout * 1e3) * 1e-3
selectors.py(464):             max_ev = max(len(self._fd_to_key), 1)
selectors.py(466):             ready = []
selectors.py(467):             try:
selectors.py(468):                 fd_event_list = self._selector.poll(timeout, max_ev)

Now I can make some HTTP requests against it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803756495 https://github.com/simonw/datasette/issues/1268#issuecomment-803756495 https://api.github.com/repos/simonw/datasette/issues/1268 MDEyOklzc3VlQ29tbWVudDgwMzc1NjQ5NQ== simonw 9599 2021-03-22T04:46:04Z 2021-03-22T04:46:04Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703  
803755698 https://github.com/simonw/datasette/issues/1249#issuecomment-803755698 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc1NTY5OA== simonw 9599 2021-03-22T04:43:02Z 2021-03-22T04:43:02Z OWNER

I'll spin off a separate ticket to investigate the hang.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803754226 https://github.com/simonw/datasette/issues/1267#issuecomment-803754226 https://api.github.com/repos/simonw/datasette/issues/1267 MDEyOklzc3VlQ29tbWVudDgwMzc1NDIyNg== simonw 9599 2021-03-22T04:37:26Z 2021-03-22T04:37:26Z OWNER

Thanks for doing this - I've used alternativeto.net a bunch in the past, it's great to see Datasette listed there.

This does raise some interesting philosophical questions: three years into the project I'm still not entirely sure what Datasette competes with! Could be SQLite desktop packages, could be visualization software like Tableau, could even be something like Airtable (given a few more plugins).

It will be interesting to see how the alternativeto listing evolves, maybe it will help me answer that question!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update Datasette alternativeto listening with details 837208901  
803753388 https://github.com/simonw/datasette/issues/1249#issuecomment-803753388 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc1MzM4OA== simonw 9599 2021-03-22T04:34:20Z 2021-03-22T04:35:10Z OWNER

Well this is frustrating. I finally found a Dockerfile that worked and installed an Ubuntu pre-compiled SpatiaLite module that would load...

FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM ubuntu:20.10

RUN apt update && \
    apt install -y python3-pip && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD ["datasette"]

(Which produced a 550MB image)

And when I ran Datasette I got that same error where the database listing page hangs!

docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite --load-extension=spatialite
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803751068 https://github.com/simonw/datasette/issues/1249#issuecomment-803751068 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc1MTA2OA== simonw 9599 2021-03-22T04:26:45Z 2021-03-22T04:26:45Z OWNER

Here's why:

datasette % docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash                                                      
root@3430352ff378:/# datasette
bash: /usr/local/bin/datasette: /usr/bin/python3: bad interpreter: No such file or directory
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803750617 https://github.com/simonw/datasette/issues/1249#issuecomment-803750617 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc1MDYxNw== simonw 9599 2021-03-22T04:25:14Z 2021-03-22T04:25:14Z OWNER

Got this error attempting to run Datasette (with or without SpatiaLite):

standard_init_linux.go:219: exec user process caused: no such file or directory
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803750399 https://github.com/simonw/datasette/issues/1249#issuecomment-803750399 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc1MDM5OQ== simonw 9599 2021-03-22T04:24:25Z 2021-03-22T04:24:25Z OWNER

I'll try using ubuntu:20.10 for everything:

FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM ubuntu:20.10 as build

RUN apt update && \
    apt install -y python3-pip && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM ubuntu:20.10

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD ["datasette"]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803749831 https://github.com/simonw/datasette/issues/1249#issuecomment-803749831 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc0OTgzMQ== simonw 9599 2021-03-22T04:22:35Z 2021-03-22T04:22:35Z OWNER

I tried copying just the mod_spatialite.so file:

FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM python:3.9.2-slim as build

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM python:3.9.2-slim

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD ["datasette"]

But when I ran Datasette with --load-extension=spatialite I got this:

  File "/usr/local/lib/python3.9/site-packages/datasette/database.py", line 151, in in_thread
    self.ds._prepare_connection(conn, self.name)
  File "/usr/local/lib/python3.9/site-packages/datasette/app.py", line 502, in _prepare_connection
    conn.execute(f"SELECT load_extension('{extension}')")
sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803748469 https://github.com/simonw/datasette/issues/1249#issuecomment-803748469 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc0ODQ2OQ== simonw 9599 2021-03-22T04:17:51Z 2021-03-22T04:17:51Z OWNER

... except my clever image using SpatiaLite installed for Ubuntu doesn't actually work:

datasette % docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
  File "/usr/local/lib/python3.9/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /usr/lib/x86_64-linux-gnu/libsqlite3.so.0)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803748158 https://github.com/simonw/datasette/issues/1249#issuecomment-803748158 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc0ODE1OA== simonw 9599 2021-03-22T04:16:57Z 2021-03-22T04:16:57Z OWNER

Which is great, because the image on Docker Hub right now is 383MB.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803747701 https://github.com/simonw/datasette/issues/1249#issuecomment-803747701 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzc0NzcwMQ== simonw 9599 2021-03-22T04:15:40Z 2021-03-22T04:15:40Z OWNER

Here's a trick: install SpatiaLite in ubuntu:20.10 and then copy it into the final python:3.9.2-slim image.

FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM python:3.9.2-slim as build

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM python:3.9.2-slim

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD ["datasette"]

That produced a 265MB image.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803700940 https://github.com/simonw/datasette/issues/1249#issuecomment-803700940 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzcwMDk0MA== simonw 9599 2021-03-22T01:14:24Z 2021-03-22T01:14:24Z OWNER

I tried that with just python3-pip (removing libsqlite3-mod-spatialite) and got 435MB.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803700626 https://github.com/simonw/datasette/issues/1249#issuecomment-803700626 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzcwMDYyNg== simonw 9599 2021-03-22T01:13:04Z 2021-03-22T01:13:04Z OWNER

Building a Dockerfile containing just FROM ubuntu:20.10 gave me 79.5MB.

Building this one:

FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && \
    apt install -y python3-pip libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

Resulted in a 515MB image.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803698983 https://github.com/simonw/datasette/issues/1249#issuecomment-803698983 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5ODk4Mw== simonw 9599 2021-03-22T01:05:36Z 2021-03-22T01:06:23Z OWNER

It's pretty big though. I tried this version which avoids copying junk from my laptop in:

FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

RUN pip install datasette

EXPOSE 8001
CMD ["datasette"]

And got this:

datasette % docker images datasette-spatialite    
REPOSITORY             TAG       IMAGE ID       CREATED         SIZE
datasette-spatialite   latest    0796950653c2   2 seconds ago   528MB
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
803698168 https://github.com/simonw/datasette/issues/1249#issuecomment-803698168 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwMzY5ODE2OA== simonw 9599 2021-03-22T01:02:02Z 2021-03-22T01:02:30Z OWNER

This is the shortest Dockerfile that appeared to give me a working SpatiaLite module:

FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

# Add local code to the image instead of fetching from pypi.
COPY . /datasette

RUN pip install /datasette

RUN rm -rf /datasette

EXPOSE 8001
CMD ["datasette"]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);