How can I figure out exactly what ways (roads, tracks) have changed based on an OSC OsmChange file?

[rowanfreeman-acutro]

I'm working on a solution to acquire, and keep updated, OSM data in a Postgres (PostGIS) database.

I'm using currently using osm2pgsql to import data from Geofabrik.

In order to keep this data up-to-date I am using Pyosmium (a Python wrapper around Osmium) to download ChangeFiles in an OSC file format (which is just XML).

Osm2pgsql can then read the change files and perform updates.

The challenge that I'm facing is that I also need to raise events based on what ways have changed. Basically, I need to figure out exactly what roads/tracks have just been altered in order to notify downstream systems. The downstream system needs to be notified about changed roads/tracks because it needs to reprocess the data for our internal business needs.

The problem is that I'm not sure how to determine exactly what has changed. From reading around, the OSC/XML ChangeFiles are not actually a good choice for this purpose. The reason for this, as I understand, is that a node could change and that node is part of a way but the way itself has not changed. For example, if the shape of an off-road track has been changed because the lat/long of a node has changed, then the ChangeFile will include the node, but not the way. This means that I can't raise an event about the way based off the OSC file.

According to this answer and this answer, they tend to agree that the OSC files don't include enough context about what has really changed.

Then I wondered if I can use osm2pgsql's processing callbacks. But the problem I noticed is that, according to the documentation:

These functions are called for each new or modified OSM object in the input file.

Well we've already established that the input file (OSC OsmChange file) doesn't have enough information and context.

So my question is: Based on an OSC OsmChange file that is about to be given to osm2pgsql, how can I determine exactly which roads/tracks are about to be updated, so that I can raise events/notifications for downstream systems?

[pnorman]

The transform functions are called for each object that was changed as a result of the OSC file, not just the objects that are changed in the OSC file.

One issue you might have is objects that are processed are potentially changed and may have stayed the same. It also calls the functions on the new/modified object, not on the change itself. Deletes never hit the transforms, and a modify is a delete/create so it looks exactly like a create.

You could also consider triggers on your feature tables but you'll have similar issues with modifies being create/delete

[ImreSamu]

my answer :

• https://community.openstreetmap.org/t/how-can-i-reliably-get-all-changed-updated-ways-based-on-an-osc-changefile/110766/3

TLDR version:

using 2x "osmium getparents" ( https://docs.osmcode.org/osmium/latest/osmium-getparents.html )
draft code:

osmium cat -f opl 004.osc.gz | cut -d' ' -f1 > changes00.id
osmium getparents hungary-latest.osm.pbf --add-self --id-file changes00.id -f opl | cut -d' ' -f1 > changes01.id
osmium getparents hungary-latest.osm.pbf --add-self --id-file changes01.id -f opl | cut -d' ' -f1 > changes02.id

and reading to postgres ...

Managing false positive changes, such as changes to a 'fixme' tag, is more complex.

[joto]

Osm2pgsql already has the logic to detect what kind of changes need to be done based on the changes of the diff file. You don't want to reimplement that. Unfortunately there is currently no good support in osm2pgsql for your use case where osm2pgsql isn't doing all the processing but some extra processing needs to be done. But you are not the first one who asked for something like this. Someday we'll find somebody who is willing to pay to build better support for that into osm2pgsql, but until that happens you can do something like this:

Create a table in osm2pgsql that you fill with data as normal. Everthing that's in there you need to run through your postprocessing. Delete everything in the table after that. Then when a change comes in you'll get new entries in that table that you have to postprocess again. You also have to create a trigger in the database that triggers on DELETE of that table and changes the DELETE into an INSERT of a second table. You can use the contents of that table to trigger deletions in your postprocessing tool chain. I you want to detect actual changes instead of treating a change as a delete plus insert, it becomes a bit more complicated. You can also do this for many tables of course and other variants are possible.