{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030902158", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030902158, "node_id": "IC_kwDOCGYnMM49clGO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T19:53:54Z", "updated_at": "2022-02-06T19:53:54Z", "author_association": "OWNER", "body": "Moving the design of this new `Conversion` subclass mechanism to:\r\n- https://github.com/simonw/sqlite-utils/issues/402", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030871591", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030871591, "node_id": "IC_kwDOCGYnMM49cdon", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T16:57:22Z", "updated_at": "2022-02-06T16:57:22Z", "author_association": "OWNER", "body": "I wonder if I could implement the above such that this *also* works:\r\n```python\r\ndb[\"places\"].insert(\r\n    {\r\n        \"name\": \"London\",\r\n        \"point\": LongitudeLatitude(-0.118092, 51.509865)\r\n    }\r\n)\r\n```\r\nThis feels like a very natural way to work with single inserts.\r\n\r\nThe challenge is writing the code inside `.insert_all()` such that it can handle these special objects in the input column values in addition to them being passed in `conversions=`.\r\n\r\nI'm feeling very good about this direction in general though, it feels like it takes the existing but not particularly elegant `conversions=` mechanism and upgrades it to be far more useful, while maintaining backwards compatibility.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030807433", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030807433, "node_id": "IC_kwDOCGYnMM49cN-J", "user": {"value": 6025893, "label": "chris48s"}, "created_at": "2022-02-06T10:54:09Z", "updated_at": "2022-02-06T10:54:09Z", "author_association": "NONE", "body": "> Interesting that some accept an SRID and others do not - presumably GeomFromGeoJSON() always uses SRID=4326?\r\n\r\nThe ewtk/ewkb ones don't accept an SRID is because ewkt encodes the SRID in the string, so you would do this with a wkt string:\r\n\r\n`GeomFromText('POINT(529090 179645)', 27700)`\r\n\r\nbut for ewkt it would be\r\n\r\n`GeomFromEWKT('SRID=27700;POINT(529090 179645)')`\r\n\r\nThe specs for KML and GeoJSON specify a Coordinate Reference System for the format\r\n\r\n- https://datatracker.ietf.org/doc/html/rfc7946#section-4\r\n- https://docs.opengeospatial.org/is/12-007r2/12-007r2.html#1274\r\n\r\nGML can specify the SRID in the XML at feature level e.g:\r\n\r\n```\r\n<gml:Point srsName=\"EPSG:27700\">\r\n  <gml:coordinates>529090, 179645</gml:coordinates>\r\n</gml:Point>\r\n```\r\n\r\nThere's a few more obscure formats in there, but broadly I think it is safe to assume an SRID param exists on the function for cases where the SRID is not implied by or specified in the input format.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030741289", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030741289, "node_id": "IC_kwDOCGYnMM49b90p", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-02-06T03:03:43Z", "updated_at": "2022-02-06T03:03:43Z", "author_association": "CONTRIBUTOR", "body": "> I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?\r\n\r\nOff the top of my head:\r\n\r\n- Un-nesting JSON objects into columns\r\n- Splitting arrays\r\n- Normalizing dates and times\r\n- URL munging with `urlparse`\r\n- Converting strings to numbers\r\n\r\nSome of this is easy enough with SQL functions, some is easier in Python. Maybe that's where having pre-built classes gets really handy, because it saves you from thinking about which way it's implemented.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740963", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740963, "node_id": "IC_kwDOCGYnMM49b9vj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T03:00:33Z", "updated_at": "2022-02-06T03:00:33Z", "author_association": "OWNER", "body": "Yeah, having this be a general purpose mechanism which has a few canned examples for handling geospatial stuff is a lot neater than having a mechanism for this that's exclusive to SpatiaLite.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740846", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740846, "node_id": "IC_kwDOCGYnMM49b9tu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:59:21Z", "updated_at": "2022-02-06T02:59:21Z", "author_association": "OWNER", "body": "I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740826", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740826, "node_id": "IC_kwDOCGYnMM49b9ta", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-02-06T02:59:10Z", "updated_at": "2022-02-06T02:59:10Z", "author_association": "CONTRIBUTOR", "body": "All this said, I don't think it's unreasonable to point people to dedicated tools like `geojson-to-sqlite`. If I'm dealing with a bunch of GeoJSON or Shapefiles, I need to something to read those anyway (or I need to figure out virtual tables). But something like this might make it easier to build those libraries, or standardize the underlying parts.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740771", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740771, "node_id": "IC_kwDOCGYnMM49b9sj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:58:29Z", "updated_at": "2022-02-06T02:58:29Z", "author_association": "OWNER", "body": "That example you have there is really neat - I like the idea that they can also be used to populate completely new columns that are derived from the other column inputs.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740653", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740653, "node_id": "IC_kwDOCGYnMM49b9qt", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-02-06T02:57:17Z", "updated_at": "2022-02-06T02:57:17Z", "author_association": "CONTRIBUTOR", "body": "I like the idea of having stock conversions you could import. I'd actually move them to a dedicated module (call it `sqlite_utils.conversions` or something), because it's different from other utilities. Maybe they even take configuration, or they're composable.\r\n\r\n```python\r\nfrom sqlite_utils.conversions import LongitudeLatitude\r\n\r\ndb[\"places\"].insert(\r\n    {\r\n        \"name\": \"London\",\r\n        \"lng\": -0.118092,\r\n        \"lat\": 51.509865,\r\n    },\r\n    conversions={\"point\": LongitudeLatitude(\"lng\", \"lat\")},\r\n)\r\n```\r\n\r\nI would definitely use that for every CSV I get with lat/lng columns where I actually need GeoJSON.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740570", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740570, "node_id": "IC_kwDOCGYnMM49b9pa", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:56:17Z", "updated_at": "2022-02-06T02:57:00Z", "author_association": "OWNER", "body": "Thinking about types. The type of the `conversions` parameter right now is a bit lazy:\r\n\r\n```python\r\nconversions: Optional[dict] = None,\r\n```\r\nThat becomes:\r\n```python\r\nOptional[Dict[str, Union[str, Conversion]]]\r\n```\r\nWhere `Conversion` is an abstract base class which expects implementations to have a `.sql() -> str` and a `.convert(value) -> str` method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030739566, "node_id": "IC_kwDOCGYnMM49b9Zu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:45:25Z", "updated_at": "2022-02-06T02:50:27Z", "author_association": "OWNER", "body": "Another idea - my favourite option so far:\r\n```python\r\nfrom sqlite_utils.utils import LongitudeLatitude\r\n\r\ndb[\"places\"].insert(\r\n    {\r\n        \"name\": \"London\",\r\n        \"point\": (-0.118092, 51.509865)\r\n    },\r\n    conversions={\"point\": LongitudeLatitude},\r\n)\r\n```\r\nHere `LongitudeLatitude` is a magical value which does TWO things: it sets up the `GeomFromText(?, 4326)` SQL function, and it handles converting the `(51.509865, -0.118092)` tuple into a `POINT({} {})` string.\r\n\r\nThis would involve a change to the `conversions=` contract - where it usually expects a SQL string fragment, but it can also take an object which combines that SQL string fragment with a Python conversion function.\r\n\r\nBest of all... this resolves the `lat, lon` v.s. `lon, lat` dilemma because you can use `from sqlite_utils.utils import LongitudeLatitude` OR `from sqlite_utils.utils import LatitudeLongitude` depending on which you prefer!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030738023", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030738023, "node_id": "IC_kwDOCGYnMM49b9Bn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:28:05Z", "updated_at": "2022-02-06T02:29:24Z", "author_association": "OWNER", "body": "Here's the definitive guide to `latitude, longitude` v.s. `longitude, latitude`: https://macwright.com/lonlat/\r\n\r\n> Which is right?\r\n>\r\n> Neither. This is an opinion with no right answer. Geographical tradition favors lat, lon. Math and software prefer lon, lat.\r\n\r\nI asked on Twitter here: https://twitter.com/simonw/status/1490148001569906688", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736848", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030736848, "node_id": "IC_kwDOCGYnMM49b8vQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:17:35Z", "updated_at": "2022-02-06T02:17:35Z", "author_association": "OWNER", "body": "Note that GeoJSON itself uses `(longitude, latitude)` so I should probably stick to that order here too. https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.1", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736589", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030736589, "node_id": "IC_kwDOCGYnMM49b8rN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:14:52Z", "updated_at": "2022-02-06T02:14:52Z", "author_association": "OWNER", "body": "Another idea: introduce a helper function transform pattern, something a bit like this:\r\n\r\n```python\r\n\r\ntransformer = make_transformer({\r\n  \"point\": lambda pair: \"POINT({} {})\".format(pair[1], pair[0])\r\n})\r\n\r\ndb[\"places\"].insert_all(\r\n    transformer([{\"name\": \"London\", \"point\": (51.509865, -0.118092)}])\r\n    conversions={\"point\": \"GeomFromText(?, 4326)\"},\r\n)\r\n```\r\nThe `make_transformer(...)` function builds an object that can work as a wrapping iterator, applying those transform functions to everything in the sequence that it wraps.\r\n\r\nSo the above code would handle converting `(lat, lon)` to `POINT(lon lat)` - then the `conversions=` applies `GeomFromText`.\r\n\r\nNaming is a challenge here: `.transform()` and `.convert()` and `conversions=` all have existing meanings within the `sqlite-utils` Python library.\r\n\r\nIt's also a bit of a messy way of solving this. It's not exactly a smooth API for inserting a bunch of lat/lon coordinate pairs!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736047", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030736047, "node_id": "IC_kwDOCGYnMM49b8iv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:10:18Z", "updated_at": "2022-02-06T02:10:18Z", "author_association": "OWNER", "body": "So maybe back to that earlier idea where the code introspects the table, figures out that `\"point\"` is a geometry table of type POINT, then applies the necessary conversions to the raw Python data?\r\n\r\nThat feels overly-complicated to me, especially since nothing else in the `.insert()` method currently relies on table introspection.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030735774", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030735774, "node_id": "IC_kwDOCGYnMM49b8ee", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:08:19Z", "updated_at": "2022-02-06T02:08:59Z", "author_association": "OWNER", "body": "Maybe I should leave this entirely up to documented patterns in the `conversions={}` dictionary?\r\n\r\nBut even that's not ideal for the co-ordinate case. Consider the following:\r\n\r\n```python\r\ndb[\"places\"].insert(\r\n    {\"name\": \"London\", \"point\": (51.509865, -0.118092)},\r\n    conversions={\"point\": \"GeomFromText(?, 4326)\"},\r\n)\r\n```\r\nThe challenge here is that the SpatiaLite function `GeomFromText()` expects a WKT string, which looks like this:\r\n\r\n    POINT(-0.118092 51.509865)\r\n\r\nThe existing `conversions=` mechanism doesn't support applying Python code to convert the `(lat, lon)` tuple to that value.\r\n\r\nIt doesn't even support passing a Python tuple as a `?` parameter - so I don't think I could come up with a SQL string that would do the right thing here either.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030732909", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030732909, "node_id": "IC_kwDOCGYnMM49b7xt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T01:47:06Z", "updated_at": "2022-02-06T01:47:06Z", "author_association": "OWNER", "body": "Here's an idea for an API design:\r\n```python\r\ngeojson_geometry = {} # ... GeoJSON goes here\r\ndb[\"places\"].insert(\r\n    {\"name\": \"Wales\", \"geometry\": geojson_geometry},\r\n    geojson=\"geometry\"\r\n)\r\n```\r\nThat `geojson=` parameter takes either a single column name or an iterable of column names. Any column in that list is expected to be a compatible `geometry` and the correct conversion functions will be applied.\r\n\r\nThat solves for GeoJSON, but it's a bit ugly. Should I add `wkt=` and maybe even `kml=` and `gml=` and so-on too? Definitely not, that's way too many ugly and inscrutable new parameters.\r\n\r\nMore importantly: if I want to support the following how would I do it?\r\n\r\n```python\r\ndb[\"places\"].insert(\r\n    {\"name\": \"London\", \"point\": (51.509865, -0.118092)}\r\n)\r\n```\r\nHere I want to provide a `(latitude, longitude)` pair and have it inserted correctly into a `point` column.\r\n\r\nCould do this, but again it's messy:\r\n```python\r\ndb[\"places\"].insert(\r\n    {\"name\": \"London\", \"point\": (51.509865, -0.118092)},\r\n    point=\"point\"\r\n)\r\n```\r\nAnd again, what about those `(longitude, latitude)` people?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}