Problems reading from bag file.

[brunovollmer]

---

Required Info
Camera Model	D430
Firmware Version	05.12.09.00
Operating System & Version	Ubuntu 20.02
Kernel Version (Linux Only)	5.8.0-55-generic
Platform	PC
SDK Version	2.40.0
Language	python
Segment	Robot

Issue Description

Hey everybody,

I have two very strange problems reading from a .bag file. The first problem is that if I set repeat_playback=False, playback.set_real_time(False) and use poll_for_frames() I receive some frames twice or three times before the next frame arrives. This is covered at the moment by the if clause at the end but if there is a solution to this I would prefer to avoid this.

The second problem is more severe and annoying and I can't really explain why. The code below runs inside a function that is called three times for three separate .bag files (we have a camera module containing three cameras). If I remove time.sleep(1) the second and third time would work as directly the first frameset are non and the break condition would apply. With the sleep time it works roughly 60% of the time. While the other 40% one of the files would just randomly return an empty frameset to early.

Any suggestions?

        camera_data = []

        pipe = rs.pipeline()
        cfg = rs.config()
        cfg.enable_device_from_file(path, repeat_playback=False)
        profile = pipe.start(cfg)
        colorizer = rs.colorizer()

        device = profile.get_device()
        playback = device.as_playback()

        playback.set_real_time(False)

        align = rs.align(rs.stream.color)

        depth_scale = (
            profile.get_device().first_depth_sensor().get_depth_scale())

        frame_numbers = []

        # sleep here because otherwise poll_for_frames will immediately fail
        time.sleep(1)

        pbar = tqdm(total=num_frames, initial=0)
        while True:
            frameset = pipe.poll_for_frames()

            if not frameset:
                break

            frame_nr = frameset.get_frame_number()
            timestamp = frameset.get_timestamp()

            if min_ts is None:
                min_ts = timestamp

            depth_frame = frameset.get_depth_frame()
            if infrared:
                color_frame = frameset.get_infrared_frame()
            else:
                color_frame = frameset.get_color_frame()

            frameset = align.process(frameset)
            depth_frame = frameset.get_depth_frame()

            color_frame = np.asanyarray(color_frame.get_data())

            if infrared:
                color_frame = cv2.cvtColor(color_frame, cv2.COLOR_GRAY2RGB)

            depth_frame_color = np.asanyarray(
                colorizer.colorize(depth_frame).get_data())

            depth_frame = np.asanyarray(depth_frame.get_data())
            depth_frame = depth_frame * depth_scale

            if frame_nr not in frame_numbers and timestamp >= min_ts:
                frame_numbers.append(frame_nr)
                pbar.update(1)

                camera_data.append({
                    'color': color_frame.copy(),
                    'depth': depth_frame.copy(),
                    'depth_color': depth_frame_color.copy()
                })

                if len(camera_data) == num_frames:
                    break

        pbar.close()
        pipe.stop()

[MartyG-RealSense]

Hi @brunovollmer If every frame was repeating 2 or 3 times instead of just some of them then it might be related to a Python bag reading issue in the link below that has already had a bug report filed with Intel about it.

#8949 (comment)

If only some of the frames are repeating then it may be due to a 'hiccup' during recording where the SDK goes back to the last known good frame and continues onward from that point. If the apperance of repeated frames is dependent on the playback settings that you use though instead of the repeating always occurring then the repeated frames may not be within the bag recording itself.

Whilst using poll_for_frames() in combination with a sleep period - as you are doing - is recommended for multiple camera appplications, for the purposes of bag playback I wonder if using wait_for_frames() instead would suffice. This would block until a frame is received, with no need to use a sleep period.

More information about the differences between the types of for_frames() instructions can be found in the link below.

#2422 (comment)

[brunovollmer]

Unfortunately wait_for_frames() does also have the problem of freezing once in a while. With my general experience with intelrealsense I was wondering if the problem might be related to not explicitly deleting something or similar approaches.

[brunovollmer]

Another problem that came up while setting up testing environment is that once every ~20 runs the same code dies with a segfault. Similar to #8601.

I have to admit that I'm quite frustrated with this framework especially when you think it is of a huge company like Intel.

[MartyG-RealSense]

In regard to the duplicate frames issue, a RealSense team member advises in the link below that use of poll_for_frames() should avoid duplicate frames occurring but it is up to the RealSense user to synchronize the frames.

#5107 (comment)

In a related case, a RealSense user solved their repeated frames problem by adding a frames.keep() instruction after their _for_frames() line.

#5125 (comment)

There is an example of a Python Keep() script here:

#3164 (comment)

[brunovollmer]

Regarding the segfault any suggestions?

[MartyG-RealSense]

Noting that you are using SDK 2.40.0: if your segmentation error takes the form of Segmentation fault (core dumped) and tends to occur when closing a stream or releasing frames, a fix was added in SDK 2.42.0

#8234

[brunovollmer]

Unfortunately I tried it with the newest version that is available on pip (pyrealsense2==2.45.0.3278) and the error continues.

[MartyG-RealSense]

Use of break conditions can sometimes cause errors that do not occur if a break is not included. Omitting the break can cause problems in term of an inability to catch errors though unless the error is of a type that can be caught as an exception.

[brunovollmer]

Not really sure what you want to tell me with this advice. One thing I forgot to add is that the function runs in a separate process.

[MartyG-RealSense]

I was asking how the program runs if you remove the two uses of the break condition in your script.

if not frameset:
break

if len(camera_data) == num_frames:
break

[brunovollmer]

This freezes and still segfaults. What I really don't get is where the segfault comes from as it only appears when I run the code inside a pytest unit test.

[brunovollmer]

I played around a bit more and if I run the code in the main thread than the freeze disappears but I do get a segfault once in a while.

[MartyG-RealSense]

As mentioned earlier in this discussion, there was a problem with sporadic segfaults in earlier SDK versions on Linux when closing streams or releasing frames that had a fix included in SDK 2.42.0.

[brunovollmer]

And as mentioned earlier I tested that version and all following ones and the segfault continues.

[brunovollmer]

Is there any further information I can give you to make it easier to find the problem?

[brunovollmer]

After some more investigation I think I found so problem. How do you submit bug reports here? Should I just state this here or open a new issue?

Anyways if anybody has the same problem:

When reading from multiple .bag files after one another the program sometimes results in a segmentation fault. The problem only results if you don't read all the files from each bag file. My solution for now is even if I only want to access a subset I still process all frame sets that are available in the file. For now I think this is a bug that should be fixed and not a problem from my side.

[MartyG-RealSense]

Thanks very much @brunovollmer for sharing information about your solution with the RealSense community. :) I think that it may be sufficient just to leave the details here for future readers to find, as it is the kind of problem that would be difficult to write a single solution for that fits all projects that use multiple bag files.

There was a past Python case in the link below where a RealSense user was experiencing segmentation errors when looping through a list of bag files.

#6887

Within that discussion, I highlighted a C++ case that experienced segmentation errors sometimes when processing a sequence of bag files.

#6887 (comment)

A RealSense team member provided feedback in that discussion about a loop test method that worked for them and allowed them to perform 500+ executions without an error occurring.

#2693 (comment)

[brunovollmer]

I still think this looks like a bug but your call. If you need further information on this, let me know!