Merging two files, but preferring data from one of those files, and interpolating the other

[spacelama]

Say you are riding with two GPS trackers - one that's quite accurate, and records temperature, cadence, but not heart rate. And the other that's not so great on locational accuracy, but records heartrate (bonus points for coping with the fact that it locks onto a location faster than the other one and isn't as likely to completely drop out). I'd like to combine these two files, preferring data over no data, but accurate data over inaccurate data. The timestamp is accurate in both, so you should just be able to interpolate the non lat,lon,time values from the second file into the first file to result in the merged file (and keeping lat,lon from the second file when it's not present in the first file).

Simply merging the files and keeping all data points does not work because each data point is likely to have only one of (time,inaccurate location,heart rate) or (time, accurate location, temperature, cadence). And your track will bounce back and forth between those two types of data, yielding quite the mess. I tested this with an invocation that looked like:
gpsbabel -t -i gpx -f Evening_Ride-20240208.gpx -i gpx -f Evening_Ride-20240208-2.gpx -x track,merge,title="COMBINED LOG" -o gpx -F Evening_Ride-20240208-combined.gpx and confirmed it was not really usable without lots of manual gpspruneing (and then you're throwing away data that was still present in the file that was ignored in any given segment).

I think I can contrive up an interpolation of both files at 1 second accuracy, and brute force the two files together ignoring those columns I want to discard from the second file, but that will involve large temporary files and a loss of accuracy if the initial files have sample points more often than once a second.

A general filter to do this would be much more useful, since it seems to be a common requirement, eg: https://help.openstreetmap.org/questions/54463/merge-gpx-tracks-without-losing-data

[robertlipe]

Our members of the list surprise me with the various solutions they come up with, but I have nothing...that'd result in a respectable map with the resolution you'd want for HR and cadence. Just averaging lat/lons doesn't work as at 1hz, moving a point even tens of feet is going to scrooch instantaneous cadence, for example. Even with both units at 1hz, they're not going to be the same samples, so you can't really just build a giant associative array and then run a thread through them. I can't think of ways that are mathematically sound to merge them. You even hinted at this. "I can make two spreadsheets and with divine knowledge about which columns are sub-awesome in each one, produce something that may involve a loss of accuracy" is the non-programmer's way of saying "write a custom filter". That said, we're open source. Writing a new filter - even one that's "Tim's special Track Wrangler" that does a crazy, but custom, thing for your own personal use - is an option you have. Internally we "just" decompose tracks into arrays of sequenced points. The data structures are a little more strict than spreadsheet rows, but if you want to build up your own thing, that's totally possible. What works for your use of your data isn't necessarily viable for others, though, which is the case we have to think about. See that somewhat comical, "LOL, no, don't you EVEN contaminate my data store with this mess of a track" response from the OSM SME in your referenced message. :-)

…

On Sat, Feb 10, 2024 at 8:06 AM Tim Connors ***@***.***> wrote: Say you are riding with two GPS trackers - one that's quite accurate, and records temperature, cadence, but not heart rate. And the other that's not so great on locational accuracy, but records heartrate (bonus points for coping with the fact that it locks onto a location faster than the other one and isn't as likely to completely drop out). I'd like to combine these two files, preferring data over no data, but accurate data over inaccurate data. The timestamp is accurate in both, so you should just be able to interpolate the non lat,lon,time values from the second file into the first file to result in the merged file (and keeping lat,lon from the second file when it's not present in the first file). Simply merging the files and keeping all data points does not work because each data point is likely to have only one of (time,inaccurate location,heart rate) or (time, accurate location, temperature, cadence). And your track will bounce back and forth between those two types of data, yielding quite the mess. I tested this with an invocation that looked like: gpsbabel -t -i gpx -f Evening_Ride-20240208.gpx -i gpx -f Evening_Ride-20240208-2.gpx -x track,merge,title="COMBINED LOG" -o gpx -F Evening_Ride-20240208-combined.gpx and confirmed it was not really usable without lots of manual gpspruneing (and then you're throwing away data that was still present in the file that was ignored in any given segment). I think I can contrive up an interpolation of both files at 1 second accuracy, and brute force the two files together ignoring those columns I want to discard from the second file, but that will involve large temporary files and a loss of accuracy if the initial files have sample points more often than once a second. A general filter to do this would be much more useful, since it seems to be a common requirement, eg: https://help.openstreetmap.org/questions/54463/merge-gpx-tracks-without-losing-data — Reply to this email directly, view it on GitHub <#1260>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD37GPH4XWXBZOA5K2ZLYS55GBAVCNFSM6AAAAABDCXZFYKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEZDQNBXGU3DONQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[tsteven4]

In the example you cite one method would be to augment the track with accurate position data with heart rate data from the other track.
Some sound methods to do this are:

1. use a zero order hold on the heart rate track and resample it at the times in the position track. Effectively this means to use the most recently available heart rate data for each point in the position track.
2. use a first order hold on the hear rate track and resample it at the times in the position track. Effectively this is just a linear interpolation of the adjacent heart rate points at each point in the position track.
3. resample the heart rate data using the resample filter, and then to 1. or 2. Note the resample filter does not currently work with heart rate, but could be made to do so.