Shapefile output seems to produce empty files?

[blackmad]

Hi there,

I'm trying to extract administrative boundary polygons from osm and osmjs looks like the best tool that knows about repairing all sorts of polygon issues. But it seems like it's producing empty shapefiles:

-rw-rw-r-- 1 blackmad blackmad 161 Mar 15 14:44 shp-osm_admin_2.osm.dbf
-rw-rw-r-- 1 blackmad blackmad 6 Mar 15 14:44 shp-osm_admin_2.osm.pbf.cpg
-rw-rw-r-- 1 blackmad blackmad 144 Mar 15 14:44 shp-osm_admin_2.osm.pbf.prj
-rw-rw-r-- 1 blackmad blackmad 100 Mar 15 14:44 shp-osm_admin_2.osm.shp
-rw-rw-r-- 1 blackmad blackmad 100 Mar 15 14:44 shp-osm_admin_2.osm.shx

I'm attempting to generate these by first isolating only the data I would need per level to make osmjs' job easier:
~blackmad/osm/bin/osmosis
--read-pbf {0}
--tf accept-relations admin_level={1}
--tf accept-relations boundary=administrative
--used-way
--used-node
--write-pbf

and then invoking osmjs like so:
osmjs -d -m -2 -l sparsetable -r -j shapefile.js shp-osm_admin_2

with shapefile.js being the demo code from the readme:
shapefile(argv[0]).
type(POLYGON).
column('id', STRING, 12).
column('type', STRING, 32).
column('admin_level', INTEGER).
column('name', STRING, 100);

is there anything I'm doing wrong here?

thanks,
--david

[joto]

You have to add "-i osm2shape" so it brings in the js/osm2shape.js library and you have to define rules for matching the OSM data to the shape format. Re-read the README and you'll find it in there.

[blackmad]

thanks the quick reply, I misunderstood needing to add the rules. I'll try to make some edits to the README to make it clearer. Much appreciated.

[blackmad]

okay, so I got pretty far, but then hit a point processing admin_level=6 where osmjs hasn't produced any output to stdout, or modified the output shapefile that it's working on in 16+ hours. I'm ... assuming this isn't normal? anything I can do here to better understand what's happening?

[joto]

Sounds like a bug in osmium. We know the multipolygon code is not perfect, there were other reports of errors. If you can post more information on what you did exactly, including all relevant files and command lines, I might be able to find the problem and fix it.
But all Osmium development has now switched to the new version at https://github.com/osmcode/libosmium, so I am not sure the old version will be fixed. (Unfortunately the multipolygon code in new version isn't ready yet either. So any solution may take some time.)
I am reopening this ticket, because it looks like there is an actual Osmium bug in there somewhere...

[blackmad]

thanks so much for offering to look.

input:
http://dump.blackmad.com/osm_admin_6.osm.pbf

commandline:
osmjs -i osm2shape -d -m -2 -l disk -j shapefile.js osm_admin_6.osm.pbf shp-osm_admin_6.osm.pbf

script:

[blackmad@ns4005169 osm-boundaries (master)]$ cat shapefile.js
shapefile(argv[0]).
type(POLYGON).
column('id', STRING, 12).
column('level', INTEGER).
column('name', STRING, 100);

area('boundary', 'administrative').
output(argv[0]).
attr('level', 'admin_level').
attr('name');

[joto]

I can confirm the problem. After a while the CPU is running at 100% but no progress. If I extract the relation where it hangs and run it separately, everything works. Strange...

[blackmad]

am I understanding the complete_ring algorithm correctly: If it finds an open ring, it then tries to attach unused, unrelated ways, until the ring closes, or it has gone through factorial(unused_ways) combinations to see if adding ways could close it. So if you extract the relation, it doesn't have as many unused ways, and completes by dropping the ring/polygon.

running gdb on it, it looks like complete_ring is recursing ~50 deep, is that normal? Should there be a max depth on how far complete_ring can recurse? How deep does it normally go?

[woodpeck]

If it does what you say then that is a bug and one that, I presume, would have been found in the mean time. It is supposed to consider only those ways that are members of the relation, and not "unused, unrelated ways".

[blackmad]

makes sense, I quickly read through the code, going to keep staring at it

On Tue, Mar 18, 2014 at 8:37 PM, Frederik Ramm [email protected]:

If it does what you say then that is a bug and one that, I presume, would
have been found in the mean time. It is supposed to consider only those
ways that are members of the relation, and not "unused, unrelated ways".

Reply to this email directly or view it on GitHubhttps://github.com//issues/87#issuecomment-38006198
.