Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Vector subscript out of range #231

Closed
DiabloRusso opened this issue Jan 30, 2017 · 33 comments
Closed

Error: Vector subscript out of range #231

DiabloRusso opened this issue Jan 30, 2017 · 33 comments

Comments

@DiabloRusso
Copy link

Hi!
I'm trying mapnik (branch v3.0.x) and mapnik-vector-tile (v1.2.2). I build x64 debug configuration for Windows. My test code:

mapnik::datasource_cache::instance().register_datasource(mapnik_dir + "/mapnik/input/shape.input");
mapnik::Map m(256, 256);
load_map(m, "style.xml");	
m.zoom_all();
mapnik::vector_tile_impl::processor ren(m);
mapnik::vector_tile_impl::merc_tile tile(0, 0, 0, 1024);
ren.update_tile(tile);

My simplified mapnik style:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE Map[]>
<Map srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over" background-color="#b5d0d0">

<Parameters>
  <Parameter name="scale">1</Parameter>
  <Parameter name="metatile">2</Parameter>
  <Parameter name="name"><![CDATA[OpenStreetMap Carto]]></Parameter>
  <Parameter name="description"><![CDATA[A general-purpose OpenStreetMap mapnik style, in CartoCSS]]></Parameter>
  <Parameter name="bounds">-180,-85.05112877980659,180,85.05112877980659</Parameter>
  <Parameter name="center">0,0,4</Parameter>
  <Parameter name="format">png</Parameter>
  <Parameter name="minzoom">0</Parameter>
  <Parameter name="maxzoom">22</Parameter>
</Parameters>
<Style name="world" filter-mode="first">
  <Rule>
    <MinScaleDenominator>750000</MinScaleDenominator>
    <PolygonSymbolizer fill="#f2efe9" />
  </Rule>
</Style>
<Layer name="world"
  minimum-scale-denominator="750000"
  srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over">
    <StyleName>world</StyleName>
    <Datasource>
       <Parameter name="file"><![CDATA[Path to\simplified_land_polygons.shp]]></Parameter>
       <Parameter name="type"><![CDATA[shape]]></Parameter>
    </Datasource>
  </Layer>

</Map>

I get an error like this? call stack:

>	test.exe!std::vector<ClipperLib::OutRec * __ptr64,std::allocator<ClipperLib::OutRec * __ptr64> >::operator[](unsigned __int64 _Pos) Line 1237	C++
 	test.exe!ClipperLib::Clipper::AppendPolygon(ClipperLib::TEdge * e1, ClipperLib::TEdge * e2) Line 2735	C++
 	test.exe!ClipperLib::Clipper::AddLocalMaxPoly(ClipperLib::TEdge * e1, ClipperLib::TEdge * e2, const mapnik::geometry::point<__int64> & Pt) Line 2267	C++
 	test.exe!ClipperLib::Clipper::ProcessHorizontal(ClipperLib::TEdge * horzEdge) Line 3103	C++
 	test.exe!ClipperLib::Clipper::ProcessHorizontals() Line 2881	C++
 	test.exe!ClipperLib::Clipper::ProcessEdgesAtTopOfScanbeam(const __int64 topY) Line 3434	C++
 	test.exe!ClipperLib::Clipper::ExecuteInternal() Line 1952	C++
 	test.exe!ClipperLib::Clipper::Execute(ClipperLib::ClipType clipType, ClipperLib::PolyTree & polytree, ClipperLib::PolyFillType subjFillType, ClipperLib::PolyFillType clipFillType) Line 1907	C++
 	test.exe!mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor>::operator()(mapnik::geometry::polygon<__int64,mapnik::geometry::rings_container> & geom) Line 292	C++
 	test.exe!mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> >::operator()(const mapnik::geometry::polygon<double,mapnik::geometry::rings_container> & geom) Line 238	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 313	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::geometry_empty,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::variant<mapnik::geometry::geometry_empty,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::visit<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void>(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 865	C++
 	test.exe!mapnik::util::apply_visitor<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double> >(mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f, const mapnik::geometry::geometry<double> & v) Line 43	C++
 	test.exe!mapnik::vector_tile_impl::detail::create_geom_layer(mapnik::vector_tile_impl::tile_layer & layer, double simplify_distance, double area_threshold, mapnik::vector_tile_impl::polygon_fill_type fill_type, bool strictly_simple, bool multi_polygon_union, bool process_all_rings) Line 161	C++
 	test.exe!mapnik::vector_tile_impl::processor::update_tile(mapnik::vector_tile_impl::tile & t, double scale_denom, int offset_x, int offset_y) Line 311	C++
 	test.exe!main(int argc, char * * argv) Line 43	C++

This is a bug or am I doing something wrong?
(ClipperLib is here)
Thanx!

@flippmoke
Copy link
Member

I am not 100% certain with out digging deep into this problem -- and as the bug is within the ClipperLib (which we are in the process of replacing), I am honestly not really tempted to currently dig into this problem too deep.

Can you first try with some other data --

Also I would use the default extents as well here:

mapnik::vector_tile_impl::merc_tile tile(0, 0, 0, 1024);

instead just do:

mapnik::vector_tile_impl::merc_tile tile(0, 0, 0);

@flippmoke
Copy link
Member

BTW - this is the replacement for the Angus clipper library on a branch -- using wagyu. You might try it out.

@DiabloRusso
Copy link
Author

Thanx for answer!
I try wagyu branch and get a similar error with extent=1024 and default extent too:

 	msvcp140d.dll!000007fee21e0b46()	Unknown
 	test.exe!std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<mapbox::geometry::wagyu::edge<__int64> > > >::operator++() Line 102	C++
 	test.exe!std::_Vector_iterator<std::_Vector_val<std::_Simple_types<mapbox::geometry::wagyu::edge<__int64> > > >::operator++() Line 336	C++
 	test.exe!mapbox::geometry::wagyu::next_edge_in_bound<__int64>(std::_List_iterator<std::_List_val<std::_List_simple_types<mapbox::geometry::wagyu::bound<__int64> *> > > & bnd, std::priority_queue<__int64,std::vector<__int64,std::allocator<__int64> >,std::less<__int64> > & scanbeam) Line 165	C++
>	test.exe!mapbox::geometry::wagyu::process_hot_pixel_edges_at_top_of_scanbeam<__int64>(__int64 top_y, std::priority_queue<__int64,std::vector<__int64,std::allocator<__int64> >,std::less<__int64> > & scanbeam, std::list<mapbox::geometry::wagyu::bound<__int64> *,std::allocator<mapbox::geometry::wagyu::bound<__int64> *> > & active_bounds, mapbox::geometry::wagyu::ring_manager<__int64> & rings) Line 99	C++
 	test.exe!mapbox::geometry::wagyu::build_hot_pixels<__int64>(std::deque<mapbox::geometry::wagyu::local_minimum<__int64>,std::allocator<mapbox::geometry::wagyu::local_minimum<__int64> > > & minima_list, mapbox::geometry::wagyu::ring_manager<__int64> & rings) Line 166	C++
 	test.exe!mapbox::geometry::wagyu::wagyu<__int64>::execute(mapbox::geometry::wagyu::clip_type cliptype, mapbox::geometry::multi_polygon<__int64,std::vector> & solution, mapbox::geometry::wagyu::fill_type subject_fill_type, mapbox::geometry::wagyu::fill_type clip_fill_type) Line 124	C++
 	test.exe!mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor>::operator()(mapbox::geometry::polygon<__int64,std::vector> & geom) Line 240	C++
 	test.exe!mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> >::operator()(const mapnik::geometry::polygon<double,mapnik::geometry::rings_container> & geom) Line 252	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 313	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::geometry_empty,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::variant<mapnik::geometry::geometry_empty,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::visit<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void>(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 865	C++
 	test.exe!mapnik::util::apply_visitor<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double> >(mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f, const mapnik::geometry::geometry<double> & v) Line 43	C++
 	test.exe!mapnik::vector_tile_impl::detail::create_geom_layer(mapnik::vector_tile_impl::tile_layer & layer, double simplify_distance, double area_threshold, mapnik::vector_tile_impl::polygon_fill_type fill_type, bool strictly_simple, bool multi_polygon_union, bool process_all_rings) Line 167	C++
 	test.exe!mapnik::vector_tile_impl::processor::update_tile(mapnik::vector_tile_impl::tile & t, double scale_denom, int offset_x, int offset_y) Line 316	C++
 	test.exe!main(int argc, char * * argv) Line 43	C++

I used the master branches of projects wagyu and geometry.hpp.
Thanx!

@flippmoke
Copy link
Member

@DiabloRusso that is really interesting -- is this a really massive polygon? Could you put together a good test case for this so that I might be able to repeat the problem?

@DiabloRusso
Copy link
Author

This my complete test project!
Thanks for help!

@flippmoke
Copy link
Member

@DiabloRusso okay, I have attempted to recreate this on my machine, but I am not able to do so currently. It all seems to run just fine. Unfortunately I am using OS X and not a windows machine so we might be having slight differences between the two.

@DiabloRusso
Copy link
Author

@flippmoke Could you try my xml style for generating a vector tiles on OS X? Maybe this error is independent of the OS and you reproduce her?

@flippmoke
Copy link
Member

@DiabloRusso I was using your xml file and data for my test

@DiabloRusso
Copy link
Author

@flippmoke please let me know when you get the test result.
Thanx!

@flippmoke
Copy link
Member

@DiabloRusso I used your test data and files, I was unable to reproduce your results.

@DiabloRusso
Copy link
Author

@flippmoke maybe I do not use the correct version of wagyu and geometry.hpp?

@wilhelmberg
Copy link

wilhelmberg commented Feb 9, 2017

Took a look at the provided test case:

Using the "empty" style.xml that comes with the package:

<Map>
<Style name="style">
    <Rule>
        <MarkersSymbolizer fill="red" />
    </Rule>
</Style>
</Map>
  • Visual Studio 2015 Update 3: compiles and runs the test case without any exceptions.
  • Visual Studio 2017 RC.4 completely crashes 💣 🔥 already at mapnik::Map m(256, 256);. Maybe because of some runtime differences (VS2015 vs VS2017)?

Using VS2015 with the style.xml form the first post above (referencing simplified_land_polygons.shp) exception occurs on ren.update_tile(tile);:

 	msvcp140d.dll!00007ffe6c7ffe06()	Unknown
>	test.exe!std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<mapbox::geometry::wagyu::edge<__int64> > > >::operator++() Line 103	C++
 	test.exe!std::_Vector_iterator<std::_Vector_val<std::_Simple_types<mapbox::geometry::wagyu::edge<__int64> > > >::operator++() Line 342	C++
 	test.exe!mapbox::geometry::wagyu::next_edge_in_bound<__int64>(std::_List_iterator<std::_List_val<std::_List_simple_types<mapbox::geometry::wagyu::bound<__int64> *> > > & bnd, std::priority_queue<__int64,std::vector<__int64,std::allocator<__int64> >,std::less<__int64> > & scanbeam) Line 165	C++
 	test.exe!mapbox::geometry::wagyu::process_hot_pixel_edges_at_top_of_scanbeam<__int64>(__int64 top_y, std::priority_queue<__int64,std::vector<__int64,std::allocator<__int64> >,std::less<__int64> > & scanbeam, std::list<mapbox::geometry::wagyu::bound<__int64> *,std::allocator<mapbox::geometry::wagyu::bound<__int64> *> > & active_bounds, mapbox::geometry::wagyu::ring_manager<__int64> & rings) Line 99	C++
 	test.exe!mapbox::geometry::wagyu::build_hot_pixels<__int64>(std::deque<mapbox::geometry::wagyu::local_minimum<__int64>,std::allocator<mapbox::geometry::wagyu::local_minimum<__int64> > > & minima_list, mapbox::geometry::wagyu::ring_manager<__int64> & rings) Line 166	C++
 	test.exe!mapbox::geometry::wagyu::wagyu<__int64>::execute(mapbox::geometry::wagyu::clip_type cliptype, mapbox::geometry::multi_polygon<__int64,std::vector> & solution, mapbox::geometry::wagyu::fill_type subject_fill_type, mapbox::geometry::wagyu::fill_type clip_fill_type) Line 124	C++
 	test.exe!mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor>::operator()(mapbox::geometry::polygon<__int64,std::vector> & geom) Line 240	C++
 	test.exe!mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> >::operator()(const mapnik::geometry::polygon<double,mapnik::geometry::rings_container> & geom) Line 252	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 313	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::geometry_empty,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::variant<mapnik::geometry::geometry_empty,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::visit<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void>(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 865	C++
 	test.exe!mapnik::util::apply_visitor<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double> >(mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f, const mapnik::geometry::geometry<double> & v) Line 43	C++
 	test.exe!mapnik::vector_tile_impl::detail::create_geom_layer(mapnik::vector_tile_impl::tile_layer & layer, double simplify_distance, double area_threshold, mapnik::vector_tile_impl::polygon_fill_type fill_type, bool strictly_simple, bool multi_polygon_union, bool process_all_rings) Line 167	C++
 	test.exe!mapnik::vector_tile_impl::processor::update_tile(mapnik::vector_tile_impl::tile & t, double scale_denom, int offset_x, int offset_y) Line 316	C++
 	test.exe!main(int argc, char * * argv) Line 25	C++
 	[External Code]	

and pauses here in C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\vector:

image

Values of bnd in active_bound_list.hpp:164 before the exception occurs:
image

Also ((*bnd)->next_edge) seems to be null here:

image

And (*bnd)->edges having two almost identical edges:
image

@flippmoke anything else I should look for?

@DiabloRusso
Copy link
Author

@BergWerkGIS you reproduce my bug! I use Visual Studio 2015 Update 2, if it has meaning.
Thanx!

@wilhelmberg
Copy link

@flippmoke did you try with the provided shapefile? At first glance it looks like one of those that cause trouble when being cut into tiles.

Some analysis:

λ shapeindex --index-parts simplified_land_polygons.shp
max tree depth:8
split ratio:0.55
processing simplified_land_polygons.shp
9994
length=250574
version=1000
type=5
extent:box2d(-20037508.3427892476320267,-20037508.3688470497727394,20037508.3427892476320267,18461504.1001019775867462)
 number shapes=62648
 number nodes=2328
done!

EPSG:3857 "WGS 84 / Pseudo Mercator"

  • single part features: 62631
  • multi part features: 0
  • null geometry features: 0
  • all features: 62631

BBoxes:
image

image

I tried exporting to mbtiles via Mapbox Studio Classic and that worked.
Checking geometry validity via QGIS gets stuck and crashes QGIS at some feature with fid>62.000 - still looking for the exact feature.

@wilhelmberg
Copy link

Removing the features that made QGIS hang didn't change the mapnik error.

fid 62056
image

fid 62261
image

fid 62620
image

@wilhelmberg
Copy link

Ah, I must have been blind, just realized the demo project has just Debug configuration.

@DiabloRusso did you build mapnik and all dependencies yourself, or did you use the mapnik-sdk we provide?

Our SDK is a Release build with debug symbols enabled, not a real Debug build.
You cannot mix Release and Debug, that leads to strange unexpected behavior - like the one we are seeing.

If you used the mapnik-sdk please change your project to Release and try again.

@DiabloRusso
Copy link
Author

@BergWerkGIS I modified mapbox/windows-builds to build a debug version of mapnik and all dependencies, it was not so difficult.

@flippmoke
Copy link
Member

@BergWerkGIS Yes, I built a test in mapnik-vector-tile based on the provided test data and xml in the project. I never had any of the issues you are seeing.

Next edge should be one ahead of the edge there, so it is rather odd that it would be null unless that is the same as it being the end() in this case?

I see that I accidentally wasn't initializing it in the constructor -- which I am going to fix and push to master of wagyu, but it still should have been overwritten by this:

https://github.com/mapbox/wagyu/blob/master/include/mapbox/geometry/wagyu/local_minimum_util.hpp#L302

which is call right after the edge is created.

@wilhelmberg
Copy link

I modified mapbox/windows-builds to build a debug version of mapnik and all dependencies, it was not so difficult.

@DiabloRusso Oh Wow!
The last time I tried there was no way getting everything to build Debug.
VS constantly crashed, 32GB RAM were not enough (plus lots of swap space) and some intermediate files hit some kind of limit - if I remember correctly.

Would you mind creating a PR at mapbox/windows-builds?

@wilhelmberg
Copy link

Would you mind creating a PR at mapbox/windows-builds?

@DiabloRusso And I suppose changes to mapnik/mapnik-gyp were also necessary?

@wilhelmberg
Copy link

I used the script below on a PowerShell Dev prompt from the root of the test project like so
.\linktype.ps1 .\

Looks like at least gdal and libtiff are Release.
Is gdal involved with shape file reading?

For the ones prefixed with ??? I couldn't find a way to determine their configuration.

function check($filter){

    Write-Host "`nchecking $filter";

    foreach ($item in Get-ChildItem $args[0] -Recurse -Include $filter){
        $output = dumpbin /DIRECTIVES $item.FullName;
        if ($output -like '*LIBCMT*') { #MT and MTd
            Write-Host "MT or MTd $item";
        }elseIf ($output -like '*MSVCRTD*') { #MDd
            Write-Host "MDd $item";
        } else {
            $output = dumpbin /SYMBOLS $item.FullName;
            if($output -like '*COFF SYMBOL TABLE*'){ #debug
                Write-Host "debug $item";
            } else {
                $output = dumpbin /DEPENDENTS $item.FullName;
                if($output -like '*VCRUNTIME140D.dll*'){ #debug
                    Write-Host "debug $item";
                } elseif($output -like '*VCRUNTIME140.dll*'){ #release
                     Write-Host "release $item";
                } else {
                    Write-Host "??? $item";
                }
            }
        }
    }
}

check('*.lib');
check('*.dll');
check('*.exe');

Write-Host 'finished';

checking *.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\boost_python-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\cairo-static.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\cairo.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\freetype.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\gdal_i.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\harfbuzz.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\icudt.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\icuind.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\icuucd.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\jpeg.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_chrono-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_date_time-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_filesystem-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_iostreams-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_prg_exec_monitor-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_program_options-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_regex-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_system-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_test_exec_monitor-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_thread-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_timer-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_unit_test_framework-vc140-mt-gd-1_62.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libboost_zlib-vc140-mt-gd-1_62.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libexpat.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libpng16.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libpqd.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libprotobuf-lite.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libtiff_i.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libwebp_debug_dll.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\mapnik-json.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\mapnik-wkt.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\mapnik.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\proj.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\sqlite3.lib
MDd C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\zlib.lib
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\zlibwapi.lib

checking *.dll
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\boost_python-vc140-mt-gd-1_62.dll
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\cairo.dll
release C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\gdal201.dll
??? C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\icudt56.dll
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\icuin56d.dll
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\icuuc56d.dll
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\jpeg62.dll
??? C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libexpat.dll
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libpng16.dll
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libpqd.dll
release C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libtiff.dll
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\libwebp_debug.dll
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\mapnik.dll
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\lib\zlibwapi.dll

checking *.exe
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\bin\mapnik-index.exe
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\bin\mapnik-render.exe
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\bin\protoc.exe
debug C:\mb\_TEMP\mapnik-test\mapnik-test\lib\x64\Debug\mapnik-sdk\bin\shapeindex.exe
finished

@wilhelmberg
Copy link

@flippmoke paused before execution of line 164:
image

  • there are 2 edges in (*bnd)->edges
  • (*bnd)->current_edge points to the second one
  • (*bnd)->next_edge is NULL
  • trying to look into (*bnd)->edges.end() shows an error

image

@flippmoke
Copy link
Member

Does current_edge point to the second one before or after the execution of line 162? If current_edge -- they both should be shifting together, it is fine if next_edge is equal to end() if the current_edge is equal to the last element in edges. Keep in mind that the pointer to end *end() should be null as it is not pointing to any element in edges. So if *next_edges is null it could be equal to end().

@DiabloRusso
Copy link
Author

Would you mind creating a PR at mapbox/windows-builds?

@BergWerkGIS Ok, I will create a PR during 2 days!

The last time I tried there was no way getting everything to build Debug.
VS constantly crashed, 32GB RAM were not enough (plus lots of swap space) and some intermediate files hit some kind of limit - if I remember correctly.

Is strangely, my first attempt to build mapnik I was doing on notebook with i3 processor, 4GB RAM and it took me nearly 8GB HDD

And I suppose changes to mapnik/mapnik-gyp were also necessary?

Now I can not say for sure, I don't remember that in this project I did changes. In most cases, I did changes in batch files. I will prepare my improved project and you will see all.

Thanks for testing my sample!

@wilhelmberg
Copy link

@DiabloRusso perfect!

@flippmoke video of stepping through those lines: mapnik-test.mp4.zip

@DiabloRusso
Copy link
Author

@BergWerkGIS I create PR! Will need to make further changes in mapnik/mapnik-gyp, in build.bat exactly! Also I create issue in mapbox/windows-builds

@flippmoke
Copy link
Member

I still don't see any way still that this bug is even possible in current code in wagyu, so my thought is we are having a memory corruption (likely upstream) from this code.

@DiabloRusso
Copy link
Author

@flippmoke within an hour, I'll create PR into Wagyu, I think I found an error...

@flippmoke
Copy link
Member

@DiabloRusso thanks! It is very hard to debug when I can't reproduce locally!

@DiabloRusso
Copy link
Author

@flippmoke Before the first get next_edge, it is never initialized!
Without going into essence of Wagyu, I found only one place where next_edge would may be initialized. I create PR at Wagyu, it solve my problem!
Thanx!

@flippmoke
Copy link
Member

@DiabloRusso thanks so much for hunting this down, its because next_edge is actually never used in the hot pixel portion of the code. The MSFT compiler is much more strict and noticed that we were iterating on a nullptr, while the operations would just more around unassigned memory in the gcc and clang configuration. I am rather sad that our static analysis tools didn't pick this up.

@DiabloRusso
Copy link
Author

DiabloRusso commented Feb 17, 2017

@flippmoke I just clean the bug, but has not check the quality of features in vector tiles. I notified you about my tests.
Thanx!
@BergWerkGIS thanx for test!

@springmeyer
Copy link
Contributor

Amazing collaboration on this everyone 👏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants