UI hangs at mbtiles export when Source is non WGS84/web merc projection

[andrewharvey]

In Mapbox Studio if I use a SHP file of a projection which isn't WGS84 / Web Mercator it seems to run into some issues exporting this Source to mbtiles prior to uploading to Mapbox.com. It may well be that using such a projection isn't supported but there are no warnings about this.

In fact the layer displays fine and I can even create styles from it without a problem.

I did some debugging and found that ./lib/source.js:311 was being reached as there wasn't any extent metadata present in the datasource. However, the ds.extent() as returned by the mapnik.Datasource object is [ 2035800, 5326900, 3016800, 6751400 ], which then becomes an issue down on line ./lib/source.js:323 where .min and .max between a projected coordinate system and a geographic coordinate system results in the extents becoming [ 2035800, 5326900, 180, 85.0511 ]. Which then seems to correctly throw an error that "bounds south value must be less than or equal to north" at ./lib/tilelive.js:245 but the user interface just sits there.

The two source files I used to test this are at http://tianjara.net/hosted/nzcoastline/

I'm not sure what fix you want to implement, but either way currently it just hangs at 0.0% when generating the mbtiles for upload.

[wilhelmberg]

@andrewharvey generally speaking, all projections supported by proj4, should be supported by Mapbox Studio.
However, some projections seem to be causing problems when reprojected on the fly from within Mapbox Studio.

When using shapefiles (for bigger data we recommend PostGIS) it really helps if you preprocess them before loading into Mapbox Studio.

In particular, preprocess:

• projection
• encoding
• shapeindex (only necessary for files with lots of geometries)

Here's a description on how to do it: #1200 (comment)

Mapbox PostGIS manual

---

I can replicate with NZ_Coastline_NZMG.shp (not with NZ_Coastline_WGS84.shp).
Mapbox Studio 0.2.7 64bit Windows installer version

Even just zooming or panning in Mapbox Studio source project crashes the whole application.

DetailID = 1
    Count:    1
    Exception #:  0XC0000005
    Stack:        
        ntdll!RtlAnsiStringToUnicodeString
        kernel32!HeapFree
        APPCRT140!_free_base
        mapnik_7fecdee0000!mapnik::geometry_utils::from_wkb
        mapnik_7fecdee0000!mapnik::geometry_utils::from_wkb
        mapnik_7fecdee0000!mapnik::projection::init_proj4
        mapnik_7fecdee0000!mapnik::proj_transform::proj_transform
        mapnik
        mapnik
        mapnik
        node!uv_fs_poll_stop
        ntdll!TpCallbackMayRunLong
        ntdll!RtlRealSuccessor
        kernel32!BaseThreadInitThunk
        ntdll!RtlUserThreadStart

DetailID = 1
    Count:    1
    Exception #:  0XC0000005
    Stack:        
        ntdll!RtlAnsiStringToUnicodeString
        kernel32!HeapFree
        APPCRT140!_free_base
        shape!destroy
        shape!destroy
        mapnik
        mapnik
        mapnik
        mapnik
        node!uv_fs_poll_stop
        ntdll!TpCallbackMayRunLong
        ntdll!TpDisassociateCallback
        kernel32!BaseThreadInitThunk
        ntdll!RtlUserThreadStart

[andrewharvey]

@BergWerkGIS

WGS84.shp was the one that works (so I could be sure it was the projection that was the issue). I didn't run into any crashes myself I could pan and zoom the Source.

Yeah the workaround is to reproject, but this should probably be fixed within Mapbox Studio to either work correctly with data in this projection, or warn the user first that it's not supported/what the problem is.

I forgot about shapeindex, thanks for the hint! (Hopefully when I include it in the shapefile ZIP uploaded to Mapbox for processing it's used)

[wilhelmberg]

or warn the user first that it's not supported/what the problem is.

💯% agreed, we are working on it, but the problem is intermittent and we haven't been able to exactly pinpoint the real reason.
Like, see above, two different stack traces for two consecutive crashes.

[andrewharvey]

💯% agreed, we are working on it, but the problem is intermittent and we haven't been able to exactly pinpoint the real reason.

The problem I described is repeatable and it seems from what I could determine that the reason is one line of code expects the coordinates to be in WGS84 but values are actually being provided in the native coordinate system, as I described.

[wilhelmberg]

The problem I described is repeatable and it seems from what I could determine that the reason is one line of code expects the coordinates to be in WGS84 but values are actually being provided in the native coordinate system, as I described

True, I've read your comment and appreciate your detective work, but the wrong coordinates really come from calling into proj4 via C++ in node-mapnik.

For most of the cases proj4 is able to reproject, but for some unfortunately not, and then the original coordinates are returned.

We were not yet able to pinpoint, what those some have in common and how to work around that, when calling into proj4 to do the reprojection.

If you have any idea how to fix this properly in C++, a PR is more than welcome!

[andrewharvey]

True, I've read your comment and appreciate your detective work, but the wrong coordinates really come from calling into proj4 via C++ in node-mapnik.

For most of the cases proj4 is able to reproject, but for some unfortunately not, and then the original coordinates are returned.

We were not yet able to pinpoint, what those some have in common and how to work around that, when calling into proj4 to do the reprojection.

I see, so in the ds.extent() call from ./lib/source.js you're saying these should be WGS84 rather than the same projection as the source SHP file?

Just testing out mapnik with python here...

import mapnik
ds=mapnik.Shapefile(file='NZMG.shp')
ds.envelope()
Box2d(2035800.0,5326900.0,3016800.0,6751400.0)
layer=mapnik.Layer('layer')
layer.datasource=ds
layer.srs='+proj=nzmg +lat_0=-41 +lon_0=173 +x_0=2510000 +y_0=6023150 +ellps=intl +units=m +no_defs'
layer.srs
'+proj=nzmg +lat_0=-41 +lon_0=173 +x_0=2510000 +y_0=6023150 +ellps=intl +units=m +no_defs'
layer.datasource.envelope()
Box2d(2035800.0,5326900.0,3016800.0,6751400.0)

It seems the datasource is in NZMG and it's envelope() is reported in NZMG, and the reprojection happens when you set a different layer.srs. (again just my understanding from what I've read and tried out).

Based on this within ./lib/source.js if we're getting the extent of the datasource why is it expected to be in WGS84, when it seems from the way the Mapnik API works it should be in NZMG (the projection of the datasource).

Sorry if I've missed something, just trying to nail what's actually going on here.

[springmeyer]

@andrewharvey @BergWerkGIS - thanks for the great, small testcase and the investigations.

To confirm: @andrewharvey are you running Mapbox Studio v0.2.7 or are you running latest from mb-pages/github?

My hunch is you are running from github as I can see that the bug you describe (not projecting source extents correctly into epsg:4326) likely happened in 823ffb9#diff-ae2e9ce210cf8e5e189658fb547ddc8a (which is still not in an official release). Before that change we were calling getCenterAndExtent from mapnik-omnivore here and via this require which correctly handled reprojecting the extent of a source into epsg:4326.

[andrewharvey]

To confirm: @andrewharvey are you running Mapbox Studio v0.2.7 or are you running latest from mb-pages/github?

latest from github

My hunch is you are running from github as I can see that the bug you describe (not projecting source extents correctly into epsg:4326) likely happened in 823ffb9#diff-ae2e9ce210cf8e5e189658fb547ddc8a (which is still not in an official release). Before that change we were calling getCenterAndExtent from mapnik-omnivore here and via this require which correctly handled reprojecting the extent of a source into epsg:4326.

Oh yes, before you had getExtent(ds, l.srs) which must have gotten the extent of the datasource in the layer's projection, wonder why it was changed?

[springmeyer]

capturing chat with @GretaCB - next actions:

• integrate the NZ_Coastline_NZMG.shp into a testcase for this bug - @andrewharvey okay if we add it to https://github.com/mapbox/mapnik-test-data? (@GretaCB)
• Add back reprojection of the extent from source projection to EPSG:4326 - We can pull code into studio like https://github.com/mapbox/mapnik-omnivore/blob/master/lib/shape.js#L91-L98. (@GretaCB)

[andrewharvey]

integrate the NZ_Coastline_NZMG.shp into a testcase for this bug - @andrewharvey okay if we add it to https://github.com/mapbox/mapnik-test-data? (@GretaCB)

@springmeyer Sure, it's public domain data.

Thanks.

[wilhelmberg]

FYI:
Minimal test case with this data:
https://github.com/BergWerkGIS/mapnik-proj4-crashes

---

I've noticed when adding the data in WGS84 and NZMG is doesn't align properly in Mapbox Studio.

Manually adding +towgs84=59.47,-5.04,187.44,0.47,-0.1,1.024,-4.5993 to the NZMG layer in data.yml solves this.

[GretaCB]

Fix is merged to mb-pages branch. Closing.

[wilhelmberg]

Reopening, not fully fixed until mapbox/mapnik-vector-tile#122 lands.

[springmeyer]

I've noticed when adding the data in WGS84 and NZMG is doesn't align properly in Mapbox Studio.

@BergWerkGIS ticketed separately at #1395

[springmeyer]

Closing now. Thanks @andrewharvey @GretaCB @BergWerkGIS for all the testing and fixes!

To recap:

• Mapbox Studio latest mb-pages had a temporary regression (that never impacted an official release) whereby the UI could not detect the right extent for custom data. This was fixed in Reproject extent if not mercator #1392
• node-mapnik 3.3.0 had a regression in the ability to transform the bbox for this alternative projection which is now fixed by Broken reprojection of layer extent when source != mercator mapnik-vector-tile#122. This never impacted Mapbox Studio since Studio had not yet been upgraded to this version. But it will nevertheless not be a problem when we upgrade Studio to [email protected] - tracked at Upgrade to [email protected] #1397.
• This testcase also uncovered a third issue: on windows reprojection may crash. We're tracking a fix for this in Proj4 initialization crash on windows mapnik-vector-tile#127.