[Bug]: PrePostProcessor.postprocess().convert_color()

[hzk7287]

OpenVINO Version

2024.3.0.16041-1e3b88e4e3f-releases/2024/3

Operating System

Windows System

Device used for inference

GPU

Framework

None

Model used

https://drive.google.com/drive/folders/1Na9z_qT3WC0gBUERhUK7d3EmDLGet-XB?usp=drive_link

Issue description

My model is a video interpolation model. The input of the model is two images.
My code is below:

ov::preprocess::PrePostProcessor ppp = ov::preprocess::PrePostProcessor::PrePostProcessor(model); //hzk

ppp.input(0).tensor().set_spatial_static_shape(1080, 1920);

ppp.input(0).tensor().set_element_type(ov::element::u8);
ppp.input(0).tensor().set_color_format(ov::preprocess::ColorFormat::NV12_SINGLE_PLANE);




ppp.input(1).tensor().set_spatial_static_shape(1080, 1920);

ppp.input(1).tensor().set_element_type(ov::element::u8);

ppp.input(1).tensor().set_color_format(ov::preprocess::ColorFormat::NV12_SINGLE_PLANE);


ppp.input(0).preprocess() //hzk
    .convert_color(ov::preprocess::ColorFormat::RGB)
    .resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR)
    .convert_element_type(ov::element::f32)
    .scale({ 1.0f / 255.0f,1.0f / 255.0f,1.0f / 255.0f });


ppp.input(1).preprocess() //hzk
    .convert_color(ov::preprocess::ColorFormat::RGB)
    .resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR)
    .convert_element_type(ov::element::f32)
    .scale({ 1.0f / 255.0f,1.0f / 255.0f,1.0f / 255.0f });
ppp.input(0).tensor().set_layout("NHWC");

ppp.input(1).tensor().set_layout("NHWC");

ppp.input(0).model().set_layout("NCHW");

ppp.input(1).model().set_layout("NCHW");

std::cout << "Preprocessor: " << ppp << std::endl;
setmodel = ppp.build();

It is OK now.
But there is a error when I continue using the following code

ov::preprocess::PrePostProcessor pppimage = ov::preprocess::PrePostProcessor::PrePostProcessor(setmodel); //hzk

// pppimage.output(0).tensor().set_element_type(ov::element::f32);
// pppimage.output(0).tensor().set_layout("NCHW");
// ov::preprocess::OutputModelInfo& OutputModel = pppimage.output(0).model();
// OutputModel.set_layout("NHWC");
// OutputModel.set_color_format(ov::preprocess::ColorFormat::RGB);

pppimage.output().tensor().set_layout("NCHW");
pppimage.output().model().set_layout("NCHW");
pppimage.output().postprocess()
   .convert_layout("NHWC")
    .convert_element_type(ov::element::u8)
    .convert_color(ColorFormat::NV12_SINGLE_PLANE); // wrong !
std::cout << "Postprocessor: " << pppimage << std::endl;
model = pppimage.build();

If I change NV12_SINGLE_PLANE to BGR, I still get a error.

Thank you !

Step-by-step reproduction

No response

Relevant log output

No response

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[ilya-lavrenov]

Hi @hzk7287
What error do you observe?

[hzk7287]

[ INFO ] Loading model files: video_interprolation.xml
Model name :main_graph
Number of inputs :2
Number of outputs :1
input_tensor_0 name : input
input_tensor_0 shape : [1,3,1080,1920]
input_tensor_1 name : onnx::Concat_1
input_tensor_1 shape : [1,3,1080,1920]
input_tensor_1 name : output
input_tensor_1 shape : [1,3,1080,1920]
Preprocessor: Input "input" (color NV12 (single plane)):
User's input tensor: [1,1620,1920,1], [N,H,W,C], u8
Model's expected tensor: [1,3,1080,1920], [N,C,H,W], f32
Pre-processing steps (4):
convert color (RGB): ([1,1620,1920,1], [N,H,W,C], u8, NV12 (single plane)) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB)
resize to model width/height: ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB)
convert type (f32): ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB)
scale (0.00392157,0.00392157,0.00392157): ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB)
Implicit pre-processing steps (1):
convert layout [N,C,H,W]: ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,3,1080,1920], [N,C,H,W], f32, RGB)
Input "onnx::Concat_1" (color NV12 (single plane)):
User's input tensor: [1,1620,1920,1], [N,H,W,C], u8
Model's expected tensor: [1,3,1080,1920], [N,C,H,W], f32
Pre-processing steps (4):
convert color (RGB): ([1,1620,1920,1], [N,H,W,C], u8, NV12 (single plane)) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB)
resize to model width/height: ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB)
convert type (f32): ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB)
scale (0.00392157,0.00392157,0.00392157): ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB)
Implicit pre-processing steps (1):
convert layout [N,C,H,W]: ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,3,1080,1920], [N,C,H,W], f32, RGB)

Postprocessor: Output "output":
Model's data tensor: [1,3,1080,1920], [N,C,H,W], f32
Post-processing steps (3):
convert layout [N,H,W,C]: ([1,3,1080,1920], [N,C,H,W], f32) -> ([1,1080,1920,3], [N,H,W,C], f32)
convert type (u8): ([1,1080,1920,3], [N,H,W,C], f32) -> ([1,1080,1920,3], [N,H,W,C], u8)
convert color (BGR): ([1,1080,1920,3], [N,H,W,C], u8

After I add ".convert_color(ColorFormat::NV12_SINGLE_PLANE) or .convert_color(ColorFormat:BGR). I get the following error.
Unhandled exception at 0x00007FF88B16FABC Microsoft C++ exception: ov::Exception at memory location 0x000000AA5AD4C788.
0x00007FF88B16FABC and 0x000000AA5AD4C788 ? what's wrong?

By the way, after I delete the convert_color function, it is OK ,but I do not want an image using RGB format, while I want an image using NV12 format.
I noticed that in " convert layout [N,C,H,W]: ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,3,1080,1920], [N,C,H,W], f32, RGB)" ,there is a RGB string.
But "Post-processing steps (3):
convert layout [N,H,W,C]: ([1,3,1080,1920], [N,C,H,W], f32) -> ([1,1080,1920,3], [N,H,W,C], f32)" I do not see any RGB or NV12 (single plane) string
I add try catch in this code.
I get
Source color format 'Unknown' is not convertible to 'NV12 (single plane)

Thank you very much !