Extend Recording to multiple vehicles #2861

rajat2004 · 2020-07-18T08:22:33Z

Replaces #2847 (branch got messed up)

Closes #3029
Closes #3123
Closes #3214

Other related issues - #2307

New Additions -

Info of all vehicles is recorded (with additional column VehicleName), VehicleName parameter in Cameras field to specify specific vehicle to use the camera from
Folder field in settings to change the directory to store data in
Allow Recording with no images (just pose, etc data) (I want to know how to record without imageFile. #3123)
Enabled parameter for starting recording when AirSim starts without needing to press R or call API (How to start recording when game begins? #3029) (Any naming suggestions are welcome)
Add method to write PPM file (for uncompressed images) (Invalid uncomressed images when recording from Unreal editor #3214)

Also fixes a bug when pressing Esc and Play multiple times causes requests to add in the vector and save multiple images (#1414 (comment))

Example settings -

{
    "SettingsVersion": 1.2,
    "SimMode": "Car",

    "Vehicles": {
        "Car1": {
          "VehicleType": "PhysXCar",
          "AutoCreate": true
        },
        "Car2": {
          "VehicleType": "PhysXCar",
          "AutoCreate": true,
          "X": 0, "Y": 10, "Z": 0
        }
    },
    "Recording": {
        "RecordOnMove": false,
        "RecordInterval": 1,
        "Cameras": [
            { "CameraName": "0", "ImageType": 0, "PixelsAsFloat": false, "VehicleName": "Car1", "Compress": true },
            { "CameraName": "0", "ImageType": 5, "PixelsAsFloat": false, "VehicleName": "Car1", "Compress": true },
            { "CameraName": "0", "ImageType": 0, "PixelsAsFloat": false, "VehicleName": "Car2", "Compress": true }
        ]
    }

}

Recording - 2020-07-11-11-48-51.zip

Testing -

Multiple Vehicles, with no recording settings (Scene image from each vehicle)
Multiple Vehicles, specific sensor settings (Settings above)
Single vehicle, no settings (Scene from that vehicle, former behaviour)

TODO:

Update docs
Unity implementation

Further possible additions -

Allow user to specify directory to store recordings in (Replacement for Documents)
Currently saving no images is not possible (For some reason this leads to an error in writing to file, even though the file has been created, and there are lines in it already)
- I suspect this might be a race condition for the first write, since there are no images, no rendering and saving of images is required, so writing of pose data is happening almost instantaneously, but the file to write the data in hasn't still been created.(Haven't confirmed this though, just a thought)
- Fixed (set is_ready flag at the end)

Any testing (especially of Folder on Windows) or suggestions would be great!

Came to mind when working on #2841
Related older issue #1346, Also includes the fix proposed in #1414

madratman · 2020-07-21T00:25:49Z

/azp run microsoft.AirSim

azure-pipelines · 2020-07-21T00:25:58Z

Azure Pipelines successfully started running 1 pipeline(s).

rajat2004 · 2020-07-26T20:29:43Z

The last commit fixes a bug in the PR which causes it to not save any images when VehicleName wasn't specified.
Example settings -

{
    "SettingsVersion": 1.2,
    "SimMode": "Car",
    "Recording": {
        "RecordOnMove": false,
        "RecordInterval": 1,
        "Cameras": [
            { "CameraName": "0", "ImageType": 0, "PixelsAsFloat": false, "Compress": true }
        ]
    }
}

And here, if we change Compress to false, the images are corrupted and can't be opened, which is what I had mentioned in #2683 (comment)

Edit: So images are not corrupted, I'm able to save them by using cv2.imshow to PNG files, the problem's in displaying using cv2.imshow (lol, just forgot cv2.waitKey(1)) and in recording (which is probably due to them being written as values directly without say PNG file header or something)

For compressed images, UE is doing the conversion to PNG format, and then we're just writing it as a binary file. For uncompressed, writing array values as binary doesn't work. I can see 2 options, either go for an external library such as libpng which will do the conversion and all (format's complex, so doesn't seem to be possible to write it ourselves), or use a different format such as PPM which is much simpler and can be written in a much simpler way, similar to how PFM file is being written

rajat2004 · 2020-07-26T20:35:51Z

AirLib/include/common/AirSimSettings.hpp

+        recording_setting.requests.clear();
+        // Add Scene image for each vehicle
+        for (const auto& vehicle : vehicles) {
+            recording_setting.requests[vehicle.first].push_back(ImageCaptureBase::ImageRequest(


The strange part which I'll take a look at later, vehicles map here doesn't contain a single vehicle when no vehicles are specified, but instead 3 (one for each simmode), and why I added getDefaultVehicleName, rather than using the first vehicle in the map.
Not sure why it's being done like this, shouldn't a single default vehicle be created in initializeVehicleSettings() instead of 3?

Can be seen by adding a line here, like Utils::log(vehicle.first).

Due to how ASimModeBase::setupVehiclesAndCamera() is defined, only vehicles that match the current simmode get instantiated. The only other place where the vehicles map gets read is in AirSimSettings::getVehicleSetting(), which only gets called in VehicleSimApiBase, and VehicleSimApiBase objects have a 1:1 relationship with the vehicles that get instantiated. As a result, I am confident the change to initializeVehicleSettings() to only initialize a single default vehicle is a safe change to make.

zimmy87

Thanks for submitting this PR @rajat2004! I have a few small comments mostly dealing with naming and style conventions that I'd like to see addressed before merging.

AirLib/include/common/AirSimSettings.hpp

AirLib/include/common/common_utils/Utils.hpp

Unreal/Plugins/AirSim/Source/Recording/RecordingThread.cpp

zimmy87 · 2021-02-03T17:02:23Z

Hey @rajat2004, the 2 failing checks are due to a build break caused by a different PR. The fix for this build break was committed yesterday, can you rebase this PR to pick up the fix and rerun all the checks?

Add `vehicles_specified` var to track whether Vehicles element was present in settings Also change all occurrences of simmode strings to fixed variables

Starts recording automatically when the simulation starts

Write uncompressed image as PPM file while Recording

Plus some cleanup

zimmy87

tested this locally and it looks good to me; will discuss with my team about merging

zimmy87 · 2021-03-01T16:09:11Z

AirLib/include/common/AirSimSettings.hpp

+        recording_setting.requests.clear();
+        // Add Scene image for each vehicle
+        for (const auto& vehicle : vehicles) {
+            recording_setting.requests[vehicle.first].push_back(ImageCaptureBase::ImageRequest(


Due to how ASimModeBase::setupVehiclesAndCamera() is defined, only vehicles that match the current simmode get instantiated. The only other place where the vehicles map gets read is in AirSimSettings::getVehicleSetting(), which only gets called in VehicleSimApiBase, and VehicleSimApiBase objects have a 1:1 relationship with the vehicles that get instantiated. As a result, I am confident the change to initializeVehicleSettings() to only initialize a single default vehicle is a safe change to make.

zimmy87 · 2021-03-01T17:09:30Z

Thank you for the contribution @rajat2004!

rajat2004 · 2021-03-01T17:15:23Z

Glad to help! Contributing to AirSim has definitely helped me improve my skills, and hope that this will be useful to others as well :)

Ryton · 2021-03-25T22:30:15Z

Bug report
At my end following minimal example causes corrupted png file(s) for uncompressed images,
i assume that the data is generated and stored, as it scales with the pixel size, but currently it'not stored away in correct format (and/or with missing header)?

Example settings.json
{
"SettingsVersion": 1.2,
"SimMode": "Car",
"Recording": {
"RecordOnMove": false,
"RecordInterval": 1,
"Cameras": [
{ "CameraName": "0", "ImageType": 0, "PixelsAsFloat": false, "Compress": true},
{ "CameraName": "0", "ImageType": 0, "PixelsAsFloat": false, "Compress": false}
]
}
}

Versions:
AirSim Version/#commit: tested with 1.2 and 1.4
UE/Unity version: 4.24.4
Autopilot version: N/A
OS Version: 64bit Win10 Pro
Used map "Default".

What's the issue you encountered?
When running in a standalone game window, and pressing "R", two images are generated, yet while the first is OK, the second one is corrupted.

"pngcheck -v uncompressed_corrupt_img_0_0_1616710947452891700.png" returns:
File: uncompressed_corrupt_img_0_0_1616710947452891700.png (110592 bytes)
this is neither a PNG or JNG image nor a MNG stream

rajat2004 · 2021-03-26T03:28:33Z

@Ryton This PR hasn't been included in the 1.4 release, you'll have to use the latest master

Ryton · 2021-03-29T13:16:04Z

Ah ok, thanks! i was not aware of this.
Will do then!

Ryton · 2021-04-19T15:07:00Z

Fixed now, using the latest master
AND
using the correct field name/structure in the json...
the capturesettings for the camera should NOT be directly alongside their position,

so NOT:

... ,
   "Cameras": {
        "Camera1": {
  "X": 0.135, "Y": -2.15,"Z": -4.50,
            "Width": 1920 ,
            "Height": 1200 
},

But rather, within CameraSettings:

...,
"Cameras": {
 "Camera1": {
			"CaptureSettings": [
      {
          "SensorType": 0,
          "Enabled": true,
          "Width": 1920 ,
          "Height": 1200 
	  }],
            "X": 0.135, "Y": -2.15,"Z": -4.50,

rajat2004 mentioned this pull request Jul 18, 2020

Extend Recording to multiple vehicles #2847

Closed

4 tasks

rajat2004 force-pushed the multi-vehicle-recording2 branch 2 times, most recently from b8f983d to d9593dc Compare July 19, 2020 15:51

rajat2004 mentioned this pull request Jul 25, 2020

Add Vehicle option for Subwindow settings #2841

Merged

rajat2004 force-pushed the multi-vehicle-recording2 branch from d9593dc to ca06159 Compare July 26, 2020 20:23

rajat2004 commented Jul 26, 2020

View reviewed changes

rajat2004 force-pushed the multi-vehicle-recording2 branch 3 times, most recently from a1ea180 to faee309 Compare July 27, 2020 23:44

This was referenced Aug 22, 2020

Allocate the number of drones after pressing "Play" instead of the setting in advance? #2922

Closed

Fixed Wing Aircraft #2955

Open

rajat2004 force-pushed the multi-vehicle-recording2 branch from faee309 to 6b75538 Compare September 19, 2020 17:05

rajat2004 mentioned this pull request Sep 19, 2020

How to start recording when game begins? #3029

Closed

jonyMarino added feature request multi-agent bug labels Oct 5, 2020

rajat2004 mentioned this pull request Dec 15, 2020

Invalid uncomressed images when recording from Unreal editor #3214

Closed

rajat2004 force-pushed the multi-vehicle-recording2 branch from 6b75538 to 90cec34 Compare December 17, 2020 03:37

jonyMarino linked an issue Dec 21, 2020 that may be closed by this pull request

Invalid uncomressed images when recording from Unreal editor #3214

Closed

rajat2004 force-pushed the multi-vehicle-recording2 branch 2 times, most recently from c7ff43a to ed78d6f Compare January 1, 2021 13:22

zimmy87 suggested changes Feb 1, 2021

View reviewed changes

AirLib/include/common/AirSimSettings.hpp Outdated Show resolved Hide resolved

AirLib/include/common/common_utils/Utils.hpp Outdated Show resolved Hide resolved

Unreal/Plugins/AirSim/Source/Recording/RecordingThread.cpp Show resolved Hide resolved

rajat2004 force-pushed the multi-vehicle-recording2 branch 3 times, most recently from 5ce5c9b to 0c5abba Compare February 1, 2021 19:25

rajat2004 force-pushed the multi-vehicle-recording2 branch from 0c5abba to e90f9a0 Compare February 3, 2021 18:34

rajat2004 requested a review from zimmy87 February 4, 2021 17:09

rajat2004 mentioned this pull request Feb 11, 2021

Add support for fixed external cameras #3320

Merged

11 tasks

rajat2004 added 11 commits February 27, 2021 11:10

Extend Recording to multiple vehicles

5c77a1c

Add option to set parent folder for Recording

c8fc79e

Allow recording without any images

97cb7d0

[docs] Add info about new parameters for Recording settings

85e8c81

Fix Recording when no vehicles are specified

29190c5

Add `vehicles_specified` var to track whether Vehicles element was present in settings Also change all occurrences of simmode strings to fixed variables

Add Enabled option to Recording settings

d174095

Starts recording automatically when the simulation starts

Create single default vehicle settings based on SimMode

afa1145

Utils: Add method to write PPM file

de4ee74

Write uncompressed image as PPM file while Recording

Rename writePfmFile to writePFMfile

ae8d02f

Plus some cleanup

Use stringstream to construct record line

1537a7d

Small cleanups, mark some methods as const

a1acd18

rajat2004 force-pushed the multi-vehicle-recording2 branch from 273e997 to a1acd18 Compare February 27, 2021 05:40

zimmy87 approved these changes Mar 1, 2021

View reviewed changes

zimmy87 merged commit 4a91e0c into microsoft:master Mar 1, 2021

rajat2004 deleted the multi-vehicle-recording2 branch March 1, 2021 17:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend Recording to multiple vehicles #2861

Extend Recording to multiple vehicles #2861

rajat2004 commented Jul 18, 2020 •

edited by zimmy87

Loading

madratman commented Jul 21, 2020

azure-pipelines bot commented Jul 21, 2020

rajat2004 commented Jul 26, 2020 •

edited

Loading

rajat2004 Jul 26, 2020

zimmy87 Mar 1, 2021

zimmy87 left a comment

zimmy87 commented Feb 3, 2021

zimmy87 left a comment

zimmy87 Mar 1, 2021

zimmy87 commented Mar 1, 2021

rajat2004 commented Mar 1, 2021

Ryton commented Mar 25, 2021 •

edited

Loading

rajat2004 commented Mar 26, 2021

Ryton commented Mar 29, 2021

Ryton commented Apr 19, 2021

Extend Recording to multiple vehicles #2861

Extend Recording to multiple vehicles #2861

Conversation

rajat2004 commented Jul 18, 2020 • edited by zimmy87 Loading

madratman commented Jul 21, 2020

azure-pipelines bot commented Jul 21, 2020

rajat2004 commented Jul 26, 2020 • edited Loading

rajat2004 Jul 26, 2020

Choose a reason for hiding this comment

zimmy87 Mar 1, 2021

Choose a reason for hiding this comment

zimmy87 left a comment

Choose a reason for hiding this comment

zimmy87 commented Feb 3, 2021

zimmy87 left a comment

Choose a reason for hiding this comment

zimmy87 Mar 1, 2021

Choose a reason for hiding this comment

zimmy87 commented Mar 1, 2021

rajat2004 commented Mar 1, 2021

Ryton commented Mar 25, 2021 • edited Loading

rajat2004 commented Mar 26, 2021

Ryton commented Mar 29, 2021

Ryton commented Apr 19, 2021

rajat2004 commented Jul 18, 2020 •

edited by zimmy87

Loading

rajat2004 commented Jul 26, 2020 •

edited

Loading

Ryton commented Mar 25, 2021 •

edited

Loading