Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proj4 initialization crash on windows #127

Closed
springmeyer opened this issue Jun 26, 2015 · 14 comments
Closed

Proj4 initialization crash on windows #127

springmeyer opened this issue Jun 26, 2015 · 14 comments
Milestone

Comments

@springmeyer
Copy link
Contributor

We are seeing that processes that need to reproject data will crash on windows. This has likely be happening for some time (causing crashing for Mapbox Studio windows users). This may be a mapnik core problem in how Mapnik uses proj4. But so far it has only seemed to happen in Mapbox Studio (and therefore seems related to vector tiles). So, let's track this issue here for now.

@wilhelmberg
Copy link

Solved!
Non thread safe setlocale was used in pj_init_ctx of proj4.

See also:
OSGeo/PROJ#226

Patch:
https://github.com/mapbox/windows-builds/blob/master/patches/proj-4.8.0-setlocale.diff

@springmeyer
Copy link
Contributor Author

awesome work tracking this down @BergWerkGIS!!!

@springmeyer
Copy link
Contributor Author

For the record: I ran the node export.js at the latest commit https://github.com/BergWerkGIS/mapnik-proj4-crashes/commit/15b8f30fc227d01da27b3bce0dc42a61b0181f4b and using the default install ([email protected]) but did not encounter a crash. The entire export ran and completed (16 r/s). So I wondered if I needed to use a different locale. I set my system locale to "German(Austria)" but that also did not prompt a crash. So, I paused and have not tried to replicate with latest node-mapnik built from source.

@wilhelmberg
Copy link

@springmeyer
Did you just set the GUI to German, or did you also change the number format?
I have GUI in English, but number format is German, decimal symbol: , not ..

As the proj4 code in question sets

setlocale(LC_NUMERIC

I suppose, that's what causes the problem.

LC_NUMERIC: Affects the decimal-point character in formatted input/output operations
and string formatting functions, as well as non-monetary information returned by localeconv.

@wilhelmberg
Copy link

Considering performance. Allmost 1/2 of the time is spent in shape_io::shape_io:

image

Hot path within shape_io:

image

@springmeyer
Copy link
Contributor Author

Did you just set the GUI to German, or did you also change the number format?

I did not change the number format. But now I did and I'm still not able to replicate a crash. Running node export.js against [email protected] gets to z15 no problem. Should I expect it to crash quickly or do I need to run it multiple times all the way to z16?.

Here are my new settings:

screen shot 2015-06-27 at 6 53 27 am

@springmeyer
Copy link
Contributor Author

Considering performance, majority of time spent in shapefile reading makes sense. I also see ~ 16 tiles/s on OS X, so I should be able to figure out how to speed things up via OS X profiling.

@wilhelmberg
Copy link

Number Format is behind "Additional Settings"

@wilhelmberg
Copy link

Should crash quickly. Never got beyond level 10.

@springmeyer
Copy link
Contributor Author

Number Format is behind "Additional Settings"

Confirmed my number format behind "Additional Settings" also lists , as the Decimal.

@springmeyer springmeyer reopened this Jun 29, 2015
@springmeyer
Copy link
Contributor Author

re-opening to remember to circle back here to further isolate and make sure I replicate after @BergWerkGIS is back next week.

@wilhelmberg
Copy link

tests with different proj4 x64 builds:

❌ -> crash
✅ -> no crash (didn't check validity of vtiles tough)

machine [email protected] master, proj 4.8.0 stock master, proj 4.8.0 patch master, proj 4.9.1 stock master, proj 4.9.1 patch
EC2 (vanilla Server 2012)
local W7
local W8.1
VM W7-1
VM W7-2

@wilhelmberg
Copy link

Summarizing above tests:

  • tested with VS2014 CTP4, should repeat with VS2015 RC/RTM
  • only tested with x64, should repeat tests with x86
  • did not find the real reason, why crash is triggered on my local machines and not on EC2, tried
    • different locales
    • different number formats
    • different location
    • diffrent env vars: %LANG%, %LC_NUMERIC%
  • I suggest we go with "replacing setlocale with _wsetlocale" instead of commenting, as the proj4 faq states: If a locale is in effect that modifies formatting of numbers, altering the role of commas and periods in numbers, then PROJ.4 will not work.

@wilhelmberg
Copy link

Test case is not crashing for me anymore, using current proj.4/master.

Recap:

Next steps:
Move back to a release version as soon as it is available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants