Fix potential undefined behavior and unsigned overflow

[springmeyer]

Clang++ has some sophisticated sanitizers that can be accessed by installing the latest clang++. This ticket stands to track the issued uncovered and fixes.

On OSX I installed latest clang like so and built the os x app like so:

export CXX="/opt/llvm/bin/clang++ -I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/c++/v1/"
export CC="/opt/llvm/bin/clang"
export LD_LIBRARY_PATH=/opt/llvm/lib/
export LIBRARY_PATH=/opt/llvm/lib/
export CXXFLAGS="${CLANG_FLAGS}"
export CFLAGS="${CLANG_FLAGS}"
export LDFLAGS="${CLANG_FLAGS} /opt/llvm/lib/clang/3.5.0/lib/darwin/libclang_rt.ubsan_osx.a -lc++abi"
make clean
export BUILDTYPE=Debug
make osx BUILDTYPE=${BUILDTYPE}
"./build/${BUILDTYPE}/Mapbox GL.app/Contents/MacOS/Mapbox GL"

Then I ran with CLANG_FLAGS equal to -fsanitize=unsigned-integer-overflow, and -fsanitize=undefined showing:

-fsanitize=undefined

../../src/renderer/line_bucket.cpp:160:22: runtime error: division by zero
../../src/renderer/line_bucket.cpp:161:22: runtime error: division by zero
../../include/llmr/util/pbf.hpp:100:46: runtime error: shift exponent 35 is too large for 32-bit type 'unsigned int'
../../src/renderer/painter_text.cpp:84:31: runtime error: division by zero
../../src/text/collision.cpp:191:58: runtime error: division by zero
../../src/text/collision.cpp:194:58: runtime error: division by zero
../../include/llmr/util/math.hpp:79:17: runtime error: division by zero
../../include/llmr/util/math.hpp:79:25: runtime error: division by zero
../../src/renderer/painter_text.cpp:102:58: runtime error: division by zero
../../src/renderer/painter_text.cpp:111:35: runtime error: division by zero
../../src/text/placement.cpp:95:36: runtime error: division by zero
../../src/text/collision.cpp:197:58: runtime error: division by zero
../../src/text/collision.cpp:200:58: runtime error: division by zero

-fsanitize=unsigned-integer-overflow

../../include/llmr/util/pbf.hpp:114:23: runtime error: negation of 1 cannot be represented in type 'unsigned int'
../../include/llmr/geometry/geometry.hpp:51:21: runtime error: unsigned integer overflow: 4224 + 4294963072 cannot be represented in type 'unsigned int'
../../include/llmr/geometry/geometry.hpp:52:21: runtime error: unsigned integer overflow: 4224 + 4294967168 cannot be represented in type 'unsigned int'
../../src/libtess2/priorityq.c:375:16: runtime error: unsigned integer overflow: 2016473283 * 1539415821 cannot be represented in type 'unsigned int'

[springmeyer]

closing, now tracked in #1486, #1406, #1485, and #1484

[springmeyer]

note: these errors (seen above):

../../include/llmr/util/pbf.hpp:114:23: runtime error: negation of 1 cannot be represented in type 'unsigned int'
../../include/llmr/geometry/geometry.hpp:51:21: runtime error: unsigned integer overflow: 4224 + 4294963072 cannot be represented in type 'unsigned int'
../../include/llmr/geometry/geometry.hpp:52:21: runtime error: unsigned integer overflow: 4224 + 4294967168 cannot be represented in type 'unsigned int'

were never resolved, but will be fixed once #4906 lands.