Saving frames for future processing

[Alonmeytal]

Could you guys please provide a code example for holding frames in memory for future use.
Currently, we can't hold more than 10 Frame objects in memory (Due to the frame buffer pool), and so we have to convert them to some other object.
My use case is Sending the RGB image for ML analysis and then using the Depth image to enhance the returned output. As far as I understand it, there's no documented way of getting the depth of a pixel if i've "released" the Frame object.

Would aligning the depth-RGB before I use depth.get_data() solve it?
Would the solution implemented in #956 provide a means to "get back" the Frame object?

[dorodnic]

Hi @Alonmeytal
Would the following snippet help for your use-case:

std::vector<rs2::frame> frames;
rs2::pipeline pipe;
auto profile = pipe.start();

for (auto&& sensor : profile.get_device().query_sensors())
{
    sensor.set_option(RS2_OPTION_FRAMES_QUEUE_SIZE, 0);
}

for (int i = 0; i < 100; i++)
{
    rs2::frameset data = pipe.wait_for_frames();
    frames.push_back(data.get_depth_frame());
}

for (auto&& frame : frames)
    std::cout << frame.get_timestamp() << std::endl;

(With latest code on the development branch this should save 100 frames)
The critical part is sensor.set_option(RS2_OPTION_FRAMES_QUEUE_SIZE, 0); - you are telling the SDK not to recycle frames for this sensor.
One practical limitation of this solution is that it won't work for framesets. In addition, if you are using processing blocks (align, colorize or the post-processing) you have to configure them as well (for instance color_map.set_option(RS2_OPTION_FRAMES_QUEUE_SIZE, 0);)
Ideally, we would prefer to have something like frame.detach() to remove specific frame from the pool, but it is not yet available.

[Alonmeytal]

@dorodnic first of all thanks for the quick response!

As you said the fact that this solution doesn't yet work for frameset rather limits my performance, as I would be forced keep converting the RGB image to a separate type (using python, so probably np.asanyarray as the examples show) or writing it directly to the FS instead of just pushing them to the same queue.
But I do think that it helps a lot.
Yet again, is there no way of just storing the depth data already "aligned" (as is, (X, Y) in the RGB would correlate to the value saved in the (X, Y) of the depth image) to begin with? That way I won't have to keep the depth frames, thus saving memory, and allowing myself to reuse the depth image in the future (until #956 is resolved).

[Jiloc]

Hello @Alonmeytal, did you find a workaround to save framesets?

[Alonmeytal]

@Jiloc do you in-memory or to disk?
In-memory is the problem, to-disk is "easy" if it's a one-way trip (you don't intend to load it back).
@dorodnic could you explain what the align function is doing?
What does it transform the depth "image" into? If an RGB frame is aligned with a depth frame does that mean that the X, Y in the RGB directly relates the the X, Y in the depth frame (and hence, X,Y in depth just hold the Z for that exact pixel in the RGB)?
If so I think my problem is solved (As all I need to do is align the depth and then saved it as is)

[Jiloc]

Hello @Alonmeytal , thank you for the fast reply

Actually our problem also is for in-memory, we have heavy post-processing works to do.
We are sending rs2::frameset(s) via rs2::frame_queue(s) between threads but they seem to lose lots of them

[Alonmeytal]

@Jiloc yet again, I found that copying the frame data to some sort of multidimensional array is the easiest way to go, but, yet again, you're losing the functionality that comes with the depth frame object, and of course that "casting" is time consuming (not too time consuming, but not too neglectable).

[dorodnic]

We discussed internally several ways to address this issue and came-up with something that should be convenient for everyone:
When you need to save the frame for a "long" (unbound) time, you need to call keep on it. This should work for all kinds of frames, including frame-sets.

[dorodnic]

Of course, there is no magic solution to heavy post-processing problem - if your post-processing takes more then 1000 / fps() milliseconds, you can only process limited number of frames without drops before you run out of memory.
The new feature is mainly intended for "snapshot" use-cases, things like "collect 100 frames, calculate something on all of them once and exit".

[Alonmeytal]

@dorodnic thanks for the response, seems like keep() would be a step in the right direction.
But could you please elaborate on the align functionality?

[dorodnic]

If you are using the D415, D435 or the SR300 device, the color camera is physically displaced relative to the "depth" camera. This means that pixel {x,y} in color stream will not correspond to {x,y} in the depth stream. Assuming what you want is a single RGBD image, you must find the correspondence between every two pixels and "align" the two streams.
Finding the correspondence is easy assuming one of the streams is depth - you can project 2D pixel in to 3D world using depth and then project it back to 2D pixel relative to the second camera coordinate system.
Whatever stream you chose to align to, the post condition of align is that all output frames will share that target stream resolution.

[Alonmeytal]

@dorodnic, just try to make sure I understand by using my own words:
I do have a D435 camera, and so if I use align(<left/right>) camera, grabbing the data (I intentionally mean using get_data()[X][Y]) at X, Y would return the depth for the X, Y pixel of the camera I "aligned to"?

[Jiloc]

Of course, there is no magic solution to heavy post-processing problem - if your post-processing takes more then 1000 / fps() milliseconds, you can only process limited number of frames without drops before you run out of memory.
The new feature is mainly intended for "snapshot" use-cases, things like "collect 100 frames, calculate something on all of them once and exit".

@dorodnic in our case we need to register a 30 seconds stream and process every frame as soon as possible. So, even if we don't release it in less then 1000 / fps() milliseconds, the important thing is that we don't lose any data.

keep() seems to be what we were looking for 👍

[Jiloc]

Hi @dorodnic, I am testing keep() and it works flawlessly. Now that the frames are permanent I'd like to force rs2::frame_queue to be infinite.

Since it doesn't work as rs2::sensor with sensor.set_option(RS2_OPTION_FRAMES_QUEUE_SIZE, 0);

shall I just assign the biggest uint available

rs2::frame_queue q(std::numeric_limits<unsigned int>::max())

or is there a more "elegant" way?

[dorodnic]

Hmm... Looking at the implementation (concurrency.h) there doesn't seem to be anything that would prevent you from passing some large number.
We have never tested this use case. @zivsha, perhaps in the recorder we have an unlimited queue?
What concerns me is that the parameter is uint in concurrency.h but it is passed as an int in rs.cpp...
Passing std::numeric_limits<unsigned int>::max() sounds safe.

[dorodnic]

I assume this issue is mostly resolved.
@Jiloc @Alonmeytal feel free to open a new one if there is anything else you need.
I think this has been a good addition to the SDK, we sincerely appreciate your feedback.

[fasaiph]

Hello @dorodnic ,
Where can I find documentation / examples on keep()?

[lauer2356]

Hello @dorodnic ,
Where can I find documentation / examples on keep()?

@fasaiphuathavornskul @dorodnic - are there any examples like this, using keep(), doing some processing, then moving to next frame. Python ideally.