Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add setting for overriding pixel format in capture component #4372

Merged
merged 5 commits into from
Mar 8, 2022
Merged

add setting for overriding pixel format in capture component #4372

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
41725b4
add setting for overriding pixel format
zimmy87 Feb 24, 2022
696e1d9
fix clang-format check
zimmy87 Mar 3, 2022
0a6cc90
responding to code review feedback
zimmy87 Mar 4, 2022
abda846
Update AirSimSettings.hpp
zimmy87 Mar 4, 2022
76f6dc2
Merge remote-tracking branch 'upstream/master' into fix-4339
zimmy87 Mar 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions AirLib/include/common/AirSimSettings.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,16 @@ namespace airlib
float HorzDistortionStrength = 0.002f;
};

struct PixelFormatOverrideSetting
{
int pixel_format = 0;
};

struct UnrealEngineSetting
{
std::map<int, PixelFormatOverrideSetting> pixel_format_override_settings;
};

struct CameraSetting
{
//nan means keep the default values set in components
Expand All @@ -193,6 +203,8 @@ namespace airlib
std::map<int, CaptureSetting> capture_settings;
std::map<int, NoiseSetting> noise_settings;

UnrealEngineSetting ue_setting;

CameraSetting()
{
initializeCaptureSettings(capture_settings);
Expand Down Expand Up @@ -1020,6 +1032,32 @@ namespace airlib
return gimbal;
}

static void loadUnrealEngineSetting(const msr::airlib::Settings& settings_json, UnrealEngineSetting& ue_setting)
{
Settings ue_settings_json;
if (settings_json.getChild("UnrealEngine", ue_settings_json)) {
Settings pixel_format_override_settings_json;
ue_setting.pixel_format_override_settings.clear();

for (int i = 0; i < Utils::toNumeric(ImageType::Count); i++) {
PixelFormatOverrideSetting pixel_format_setting;
pixel_format_setting.pixel_format = 0; // EXPixelformat::PF_Unknown
ue_setting.pixel_format_override_settings[i] = pixel_format_setting;
}

if (ue_settings_json.getChild("PixelFormatOverride", pixel_format_override_settings_json)) {
for (size_t child_index = 0; child_index < pixel_format_override_settings_json.size(); ++child_index) {
Settings pixel_format_child_json;
if (pixel_format_override_settings_json.getChild(child_index, pixel_format_child_json)) {
int image_type = pixel_format_child_json.getInt("ImageType", 0);
PixelFormatOverrideSetting& pixel_format_setting = ue_setting.pixel_format_override_settings.at(image_type);
pixel_format_setting.pixel_format = pixel_format_child_json.getInt("PixelFormat", 0); // default to EXPixelformat::PF_Unknown
}
}
}
}
}

static CameraSetting createCameraSetting(const Settings& settings_json)
{
CameraSetting setting;
Expand All @@ -1033,6 +1071,8 @@ namespace airlib
if (settings_json.getChild("Gimbal", json_gimbal))
setting.gimbal = createGimbalSetting(json_gimbal);

loadUnrealEngineSetting(settings_json, setting.ue_setting);

return setting;
}

Expand Down
33 changes: 20 additions & 13 deletions Unreal/Plugins/AirSim/Source/PIPCamera.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,16 +36,16 @@ APIPCamera::APIPCamera(const FObjectInitializer& ObjectInitializer)

PrimaryActorTick.bCanEverTick = true;

image_type_to_pixel_format_map_.Add(0, EPixelFormat::PF_FloatRGBA);
image_type_to_pixel_format_map_.Add(1, EPixelFormat::PF_DepthStencil); // not used. init_auto_format is called in setupCameraFromSettings()
image_type_to_pixel_format_map_.Add(2, EPixelFormat::PF_DepthStencil); // not used for same reason as above
image_type_to_pixel_format_map_.Add(3, EPixelFormat::PF_DepthStencil); // not used for same reason as above
image_type_to_pixel_format_map_.Add(4, EPixelFormat::PF_DepthStencil); // not used for same reason as above
image_type_to_pixel_format_map_.Add(5, EPixelFormat::PF_FloatRGBA);
image_type_to_pixel_format_map_.Add(6, EPixelFormat::PF_FloatRGBA);
image_type_to_pixel_format_map_.Add(7, EPixelFormat::PF_FloatRGBA);
image_type_to_pixel_format_map_.Add(8, EPixelFormat::PF_FloatRGBA);
image_type_to_pixel_format_map_.Add(9, EPixelFormat::PF_FloatRGBA);
image_type_to_pixel_format_map_.Add(Utils::toNumeric(ImageType::Scene), EPixelFormat::PF_B8G8R8A8);
image_type_to_pixel_format_map_.Add(Utils::toNumeric(ImageType::DepthPlanar), EPixelFormat::PF_DepthStencil); // not used. init_auto_format is called in setupCameraFromSettings()
image_type_to_pixel_format_map_.Add(Utils::toNumeric(ImageType::DepthPerspective), EPixelFormat::PF_DepthStencil); // not used for same reason as above
image_type_to_pixel_format_map_.Add(Utils::toNumeric(ImageType::DepthVis), EPixelFormat::PF_DepthStencil); // not used for same reason as above
image_type_to_pixel_format_map_.Add(Utils::toNumeric(ImageType::DisparityNormalized), EPixelFormat::PF_DepthStencil); // not used for same reason as above
image_type_to_pixel_format_map_.Add(Utils::toNumeric(ImageType::Segmentation), EPixelFormat::PF_B8G8R8A8);
image_type_to_pixel_format_map_.Add(Utils::toNumeric(ImageType::SurfaceNormals), EPixelFormat::PF_B8G8R8A8);
image_type_to_pixel_format_map_.Add(Utils::toNumeric(ImageType::Infrared), EPixelFormat::PF_B8G8R8A8);
image_type_to_pixel_format_map_.Add(Utils::toNumeric(ImageType::OpticalFlow), EPixelFormat::PF_B8G8R8A8);
image_type_to_pixel_format_map_.Add(Utils::toNumeric(ImageType::OpticalFlowVis), EPixelFormat::PF_B8G8R8A8);

object_filter_ = FObjectFilter();
}
Expand Down Expand Up @@ -382,19 +382,26 @@ void APIPCamera::setupCameraFromSettings(const APIPCamera::CameraSetting& camera
const auto& noise_setting = camera_setting.noise_settings.at(image_type);

if (image_type >= 0) { //scene capture components
auto pixel_format_override = camera_setting.ue_setting.pixel_format_override_settings.find(image_type);
EPixelFormat pixel_format = EPixelFormat::PF_Unknown;
if (pixel_format_override != camera_setting.ue_setting.pixel_format_override_settings.end()) {
pixel_format = static_cast<EPixelFormat>(pixel_format_override->second.pixel_format);
}
pixel_format = (pixel_format == EPixelFormat::PF_Unknown ? image_type_to_pixel_format_map_[image_type] : pixel_format);

switch (Utils::toEnum<ImageType>(image_type)) {
case ImageType::Scene:
case ImageType::Infrared:
updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], false, image_type_to_pixel_format_map_[image_type], capture_setting, ned_transform, false);
updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], false, pixel_format, capture_setting, ned_transform, false);
break;

case ImageType::Segmentation:
case ImageType::SurfaceNormals:
updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], true, image_type_to_pixel_format_map_[image_type], capture_setting, ned_transform, true);
updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], true, pixel_format, capture_setting, ned_transform, true);
break;

default:
updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], true, image_type_to_pixel_format_map_[image_type], capture_setting, ned_transform, false);
updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], true, pixel_format, capture_setting, ned_transform, false);
break;
}
setDistortionMaterial(image_type, captures_[image_type], captures_[image_type]->PostProcessSettings);
Expand Down
14 changes: 13 additions & 1 deletion docs/settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,15 @@ Below are complete list of settings available along with their default values. I
"Pitch": NaN, "Roll": NaN, "Yaw": NaN
},
"X": NaN, "Y": NaN, "Z": NaN,
"Pitch": NaN, "Roll": NaN, "Yaw": NaN
"Pitch": NaN, "Roll": NaN, "Yaw": NaN,
"UnrealEngine": {
"PixelFormatOverride": [
{
"ImageType": 0,
"PixelFormat": 0
}
]
}
},
"OriginGeopoint": {
"Latitude": 47.641468,
Expand Down Expand Up @@ -347,6 +355,10 @@ This adds fluctuations on horizontal line.
### Gimbal
The `Gimbal` element allows to freeze camera orientation for pitch, roll and/or yaw. This setting is ignored unless `ImageType` is -1. The `Stabilization` is defaulted to 0 meaning no gimbal i.e. camera orientation changes with body orientation on all axis. The value of 1 means full stabilization. The value between 0 to 1 acts as a weight for fixed angles specified (in degrees, in world-frame) in `Pitch`, `Roll` and `Yaw` elements and orientation of the vehicle body. When any of the angles is omitted from json or set to NaN, that angle is not stabilized (i.e. it moves along with vehicle body).

### UnrealEngine
This element contains settings specific to the Unreal Engine. These will be ignored in the Unity project.
* `PixelFormatOverride`: This contains a list of elements that have both a `ImageType` and `PixelFormat` setting. Each element allows you to override the default pixel format of the UTextureRenderTarget2D object instantiated for the capture specified by the `ImageType` setting. Specifying this element allows you to prevent crashes caused by unexpected pixel formats (see [#4120](https://github.com/microsoft/AirSim/issues/4120) and [#4339](https://github.com/microsoft/AirSim/issues/4339) for examples of these crashes). A full list of pixel formats can be viewed [here](https://docs.unrealengine.com/4.27/en-US/API/Runtime/Core/EPixelFormat/).

## External Cameras
This element allows specifying cameras which are separate from the cameras attached to the vehicle, such as a CCTV camera. These are fixed cameras, and don't move along with the vehicles. The key in the element is the name of the camera, and the value i.e. settings are the same as `CameraDefaults` described above. All the camera APIs work with external cameras, including capturing images, changing the pose, etc by passing the parameter `external=True` in the API call.

Expand Down