issue_comments

19 rows where issue = 324835838 sorted by updated_at descending

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

user

author_association

issue

  • Handle spatialite geometry columns better · 19
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
401312981 https://github.com/simonw/datasette/issues/276#issuecomment-401312981 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDQwMTMxMjk4MQ== russss 45057 2018-06-29T10:14:54Z 2018-06-29T10:14:54Z CONTRIBUTOR

@RusSs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg kartena/Proj4Leaflet) although the leaflet side would need to detect or be informed of the original projection?

Well, as @simonw mentioned, GeoJSON only supports WGS84, and GeoJSON (and/or TopoJSON) is the standard we probably want to aim for. On-the-fly reprojection in spatialite is not an issue anyway, and in general I think you want to be serving stuff to web maps in WGS84 or Web Mercator.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
401310732 https://github.com/simonw/datasette/issues/276#issuecomment-401310732 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDQwMTMxMDczMg== psychemedia 82988 2018-06-29T10:05:04Z 2018-06-29T10:07:25Z CONTRIBUTOR

@russs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg kartena/Proj4Leaflet) although the leaflet side would need to detect or be informed of the original projection?

Another possibility would be to provide an easy way/guidance for users to create an FK'd table containing the WGS84 projection of a non-WGS84 geometry in the original/principle table? This could then as a proxy for serving GeoJSON to the leaflet map?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
393106520 https://github.com/simonw/datasette/issues/276#issuecomment-393106520 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MzEwNjUyMA== russss 45057 2018-05-30T10:09:25Z 2018-05-30T10:09:25Z CONTRIBUTOR

I don't think it's unreasonable to only support spatialite geometries in a coordinate reference system which is at least transformable to WGS84. It would be nice to support different CRSes in the database so conversion to spatialite from the source data is lossless.

I think the working CRS for datasette should be WGS84 though (leaflet requires it, for example) - it's just a case of calling ST_Transform(geom, 4326) on the column while we're loading the data.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
393014943 https://github.com/simonw/datasette/issues/276#issuecomment-393014943 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MzAxNDk0Mw== simonw 9599 2018-05-30T02:59:53Z 2018-05-30T02:59:53Z OWNER

I just realised a problem with GeoJSON is that it assumes that the underlying geometry is WGS 84 latitude/longitude points - but it's very possible for a SpatiaLite geometry to contain geometric data that's nothing to do with geospatial projections.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
392969173 https://github.com/simonw/datasette/issues/276#issuecomment-392969173 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5Mjk2OTE3Mw== simonw 9599 2018-05-29T22:32:08Z 2018-05-29T22:32:08Z OWNER

The more time I spend with SpatiaLite the more convinced I am that this should be default behavior. There's nothing useful about the binary Geometry representation - it's not even valid WKB. I'm on board with WKT as the default display in HTML and GeoJSON as the default for .json

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
392825746 https://github.com/simonw/datasette/issues/276#issuecomment-392825746 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MjgyNTc0Ng== russss 45057 2018-05-29T15:42:53Z 2018-05-29T15:42:53Z CONTRIBUTOR

I haven't had time to look further into this, but if doing this as a plugin results in useful hooks then I think we should do it that way. We could always require the plugin as a standard dependency.

I think this is going to result in quite a bit of refactoring anyway so it's a good time to add hooks regardless.

On the other hand, if we have to add lots of specialist hooks for it then maybe it's worth integrating into the core.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
392815673 https://github.com/simonw/datasette/issues/276#issuecomment-392815673 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MjgxNTY3Mw== simonw 9599 2018-05-29T15:17:04Z 2018-05-29T15:17:04Z OWNER

I'm coming round to the idea that this should be baked into Datasette core - see above referenced issues for some of the explorations I've been doing around this area.

Datasette should absolutely work without SpatiaLite, but it's such a huge bonus part of the SQLite ecosystem that I'm happy to ship features that take advantage of it without being relegated to plugins.

I'm also becoming aware that there aren't really that many other interesting loadable extensions for SQLite. If SpatiaLite was one of dozens I'd feel that a rule that "anything dependent on an extension lives in a plugin" would make sense, but as it stands I think 99% of the time the only loadable extensions people will be using will be SpatiaLite and json1 (and json1 is available in the amalgamation anyway).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
392316306 https://github.com/simonw/datasette/issues/276#issuecomment-392316306 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MjMxNjMwNg== simonw 9599 2018-05-27T09:00:46Z 2018-05-27T09:00:46Z OWNER

Relevant to this ticket: I've been playing with a plugin that automatically renders any GeoJSON cells as leaflet maps: https://github.com/simonw/datasette-leaflet-geojson

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
392316250 https://github.com/simonw/datasette/issues/276#issuecomment-392316250 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MjMxNjI1MA== simonw 9599 2018-05-27T08:59:46Z 2018-05-27T08:59:46Z OWNER

It looks like we can use the geometry_columns table to introspect which columns are SpatiaLite geometries. It includes a geometry_type integer which is documented here: https://www.gaia-gis.it/fossil/libspatialite/wiki?name=switching-to-4.0

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
392279644 https://github.com/simonw/datasette/issues/276#issuecomment-392279644 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MjI3OTY0NA== simonw 9599 2018-05-26T18:34:21Z 2018-05-26T18:34:21Z OWNER

I've been thinking a bit about modifying the SQL select statement used for the table view recently. I've run into a few examples of SQLite database that slow to a crawl when viewed with datasette because the rows are too big, so there's definitely scope for supporting custom select clauses (avoiding some columns, showing length(colname) for others).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
392279508 https://github.com/simonw/datasette/issues/276#issuecomment-392279508 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MjI3OTUwOA== simonw 9599 2018-05-26T18:32:07Z 2018-05-26T18:32:07Z OWNER

Related: I started the documentation for using SpatiaLite with Datasette here: https://datasette.readthedocs.io/en/latest/spatialite.html

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
391505930 https://github.com/simonw/datasette/issues/276#issuecomment-391505930 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MTUwNTkzMA== russss 45057 2018-05-23T21:41:37Z 2018-05-23T21:41:37Z CONTRIBUTOR

I'm not keen on anything that modifies the SQLite file itself on startup

Ah I didn't mean that - I meant altering the SELECT query to fetch the data so that it ran a spatialite function to transform that specific column.

I think that's less useful as a general-purpose plugin hook though, and it's not that hard to parse the WKB in Python (my default approach would be to use shapely, which is great, but geomet looks like an interesting pure-python alternative).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
391504757 https://github.com/simonw/datasette/issues/276#issuecomment-391504757 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MTUwNDc1Nw== simonw 9599 2018-05-23T21:37:07Z 2018-05-23T21:37:18Z OWNER

That said, it looks like we may be able to use a library like https://github.com/geomet/geomet to run the conversion from WKB entirely in Python space.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
391504199 https://github.com/simonw/datasette/issues/276#issuecomment-391504199 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MTUwNDE5OQ== simonw 9599 2018-05-23T21:35:17Z 2018-05-23T21:35:17Z OWNER

I'm not keen on anything that modifies the SQLite file itself on startup - part of the datasette contract is that it should work with any SQLite file you throw at it without having any side-effects.

A neat thing about SQLite is that because everything happens in the same process there's very little additional overhead involved in executing extra SQL queries - even if we ran a query-per-row to transform data in one specific column it shouldn't add more than a few ms to the total page load time (whereas with MySQL all of the extra query overhead would kill us).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
391050113 https://github.com/simonw/datasette/issues/276#issuecomment-391050113 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MTA1MDExMw== russss 45057 2018-05-22T16:13:00Z 2018-05-22T16:13:00Z CONTRIBUTOR

Yup, I'll have a think about it. My current thoughts are for spatialite we'll need to hook into the following places:

  • Inspection, so we can detect which columns are geometry columns. (We also currently ignore spatialite tables during inspection, it may be worth moving that to the plugin as well.)
  • After data load, so we can convert WKB into the correct intermediate format for display. The alternative here is to alter the select SQL itself and get spatialite to do this conversion, but that strikes me as a bit more complex and possibly not as useful.
  • HTML rendering.
  • Querying?

The rendering and querying hooks could also potentially be used to move the units support into a plugin.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
391025841 https://github.com/simonw/datasette/issues/276#issuecomment-391025841 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MTAyNTg0MQ== simonw 9599 2018-05-22T15:06:36Z 2018-05-22T15:06:36Z OWNER

The other reason I mention plugins is that I have an idea to outlaw JavaScript entirely from Datasette core and instead encourage ALL JavaScript functionality to move into plugins.right now that just means CodeMirror. I may set up some of those plugins (like CodeMirror) as default dependencies so you get them from "pip install datasette".

I like the neatness of saying that core Datasette is a very simple JSON + HTML application, then encouraging people to go completely wild with JavaScript in the plugins.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
391000659 https://github.com/simonw/datasette/issues/276#issuecomment-391000659 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MTAwMDY1OQ== simonw 9599 2018-05-22T13:59:27Z 2018-05-22T13:59:27Z OWNER

Right now the plugin stuff is early enough that I'd like to get as many potential plugin hooks as possible crafted out A much easier to judge if they should be added as actual hooks if we have a working branch prototype of them.

Some kind of mechanism for custom column display is already needed - eg there are columns where I want to say "render this as markdown" or "URLify any links in this text" - or even "use this date format" or "add commas to this integer".

You can do it with a custom template but a lower-level mechanism would be nicer.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
390795067 https://github.com/simonw/datasette/issues/276#issuecomment-390795067 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MDc5NTA2Nw== russss 45057 2018-05-21T21:55:57Z 2018-05-21T21:55:57Z CONTRIBUTOR

Well, we do have the capability to detect spatialite so my intention certainly wasn't to require it.

I can see the advantage of having it as a plugin but it does touch a number of points in the code. I think I'm going to attack this by refactoring the necessary bits and seeing where that leads (which was my plan anyway).

I think my main concern is - if I add certain plugin hooks for this, is anything else ever going to use them? I'm not sure I have an answer to that question yet, either way.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
390707760 https://github.com/simonw/datasette/issues/276#issuecomment-390707760 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MDcwNzc2MA== simonw 9599 2018-05-21T16:30:35Z 2018-05-21T16:30:35Z OWNER

This probably needs to be in a plugin simply because getting Spatialite compiled and installed is a bit of a pain.

It's a great opportunity to expand the plugin hooks in useful ways though.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 26.35ms · About: github-to-sqlite