issue_comments

Useful thoughts on Twitter regarding making coordinate pairs easy and more complex shapes possible:

https://twitter.com/dbreunig/status/1490099303888547843

That is exactly where I was going: two modes.

1. Heuristics and assumptions to get coordinates as a pair (in tuple) or as columns (look for lat, lon, latitude, longitude, etc).
2. GIS mode with projections, polys, etc

Make it easy for people with csvs of coordinates. If you're using Geojson or shp files, you have to specify.

I'm trying to think of ways to make this nicer from the perspective of someone calling the .insert() or .insert_all() methods against a table that has geometry columns.

One option would be for the code to introspect the table (if it exists) before running the insert, looking for any geometry columns.

This introspection isn't easy! The table schema just gives you "name_of_column" point or similar - to figure out the SRID and suchlike you need to consult the geometry_columns table, I think - which throws a 500 error on https://calands.datasettes.com/calands/geometry_columns for some reason. Also does the shape of that table change between SpatiaLite versions?

Assuming we can introspect the table, what would we do with that information? We could add code that detects if the user attempted to pass GeoJSON objects and automatically inserts a GeomFromGeoJSON() function call - but detecting GeoJSON is a bit weird, and GeoJSON also isn't necessarily the nicest format for populating e.g. latitude/longitude points.

Maybe we just support the simplest possible case: a tuple of floats, which we assume is latitude, longitude (or should we expect longitude, latitude, the eternal debate?) - if those are used against a geometry table (especially a point table) we assume they are coordinates that need to be converted using GeomFromText('POINT(....

Not crazy about either of these ideas. Is there something better?

@eyeseast how do you usually insert geometries at the moment?

New documentation section: https://docs.datasette.io/en/latest/internals.html#datasette-tracer

Needs documentation. I'll document from datasette.tracer import trace too.

Got a prototype working with contextvars - it identified two parallel executing queries using the patch from above:

For ingesting geometry data from the command-line maybe GeoJSON would be the best route?

Wow, it was the newlines that broke it! This works fine: sql select AsWKT(SetSRID(GeomFromGeoJSON('{"type": "Point","coordinates": [-94.921875,45.460130637921004]}'), 4326)) https://calands.datasettes.com/calands?sql=select+AsWKT%28SetSRID%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%2C+4326%29%29

And removing SetSRID() returns exactly the same result:

https://calands.datasettes.com/calands?sql=select+AsWKT%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%29

I can't seem to get GeomFromGeoJSON() to work - example:

https://calands.datasettes.com/calands?sql=select+IsValid%28SetSRID%28GeomFromGeoJSON%28%27%7B%0D%0A++++++++%22type%22%3A+%22Point%22%2C%0D%0A++++++++%22coordinates%22%3A+%5B%0D%0A++++++++++-94.921875%2C%0D%0A++++++++++45.460130637921004%0D%0A++++++++%5D%0D%0A++++++%7D%27%29%2C+4326%29%29

sql select IsValid(SetSRID(GeomFromGeoJSON('{ "type": "Point", "coordinates": [ -94.921875, 45.460130637921004 ] }'), 4326)) Returns -1 suggesting the geometry is not valid. Just doing this (with or without that SetSRID() function) returns null:

sql select SetSRID(GeomFromGeoJSON('{ "type": "Point", "coordinates": [ -94.921875, 45.460130637921004 ] }'), 4326)

I wonder what the most developer-friendly way to insert geometry data into SpatiaLite is?

From https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html it looks like these are the main options:

• GeomFromText( wkt String [ , SRID Integer] )
• GeomFromWKB( wkbGeometry Binary [ , SRID Integer] )
• GeomFromKml( KmlGeometry String )
• GeomFromGML( gmlGeometry String )
• GeomFromGeoJSON( geoJSONGeometry String )
• GeomFromEWKB( ewkbGeometry String )
• GeomFromEWKT( ewktGeometry String )
• GeomFromFGF( fgfGeometry Binary [ , SRID Integer] )
• GeomFromTWKB( twkbGeometry BLOB [ , SRID Integer] )
• GeomFromGPB( geom GPKG Blob Geometry ) - GeoPackage format
• GeomFromExifGpsBlob( image BLOB )

Interesting that some accept an SRID and others do not - presumably GeomFromGeoJSON() always uses SRID=4326?

The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like {"rowid": int, "geometry": bytes}. Might be worth nudging people to explicitly create a table first, then add geometry columns.

That's a good call. I'm happy for sqlite-utils add-geometry-column to throw an error if the table doesn't exist yet.

The conversions= argument to .insert() and friends is designed to handle this case, but I don't think it's very elegant: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions

python db["places"].insert( {"name": "Wales", "geometry": wkt}, conversions={"geometry": "GeomFromText(?, 4326)"}, )

I like these designs a lot. I would suggest sqlite-utils create database.db --init-spatialite there for consistency with the sqlite-utils init-spatialite database.db command.

The other part of this story is how we support actually inserting spatial data from the command-line. I opened an issue about the challenges in doing that for the Python API here - #399 - but we need a good answer for the CLI too.

I don't yet have any good ideas here. The conversions= option in the Python library was designed to cover these kinds of cases but it's pretty clunky and I don't think it's very widely used: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions