Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SimGetImages() is extremely slow at sending even very small images #111

Closed
yannbouteiller opened this issue Nov 3, 2019 · 14 comments
Closed

Comments

@yannbouteiller
Copy link

yannbouteiller commented Nov 3, 2019

I know this problem is already well known, but obviously this is very painful for Reinforcement Learning approaches because we need to sample a huge amount of observations to make our algorithms learn anything.

On the laptop I use for testing, using SimGetImages() on one single drone with 240*320 images I get images at ~17 FPS. If I use it on both drones I drop to ~ 6 FPS (times 2 images = ~ 12 FPS).

I was hoping that using much lower resolutions would greatly improve this, but it does not. With 12*16 images for instance I get ~ 20 FPS instead of 17 with one drone, and ~ 7 FPS instead of 6 with two drones.

According to pyinstrument which I use for profiling, all this time is spent in the IOLoop of tornado:

118.958 simGetImages airsimneurips/client.py:217
│ │ [100 frames hidden] airsimneurips, msgpackrpc, tornado, l...
│ │ 118.844 start tornado/ioloop.py:753
│ │ └─ 118.760 [self]

But I suppose this just means that AirSim is super slow at computing even very small camera images.

@saihv
Copy link
Contributor

saihv commented Nov 3, 2019

It is indeed true that AirSim/Unreal are slow at image capture, but I am surprised by the very minimal gain in performance between 320x240 and 16x12. What is the configuration of your laptop - and what is your GPU usage like as you're running this? I have a feeling you're being massively bottlenecked somewhere. If you get a chance, could you perhaps try a couple of things:

  1. As you're grabbing images, can you run in the Unreal console (which you can access by pressing the tilde key ~ / ``` ) stat unitgraph and post a screenshot of the graph that appears?

  2. Can you repeat your benchmarking by locking the game framerate? You can do this by going to the console (~) and using this command : t.maxFPS 30 (I would also recommend trying 15 or 20 to evaluate)

  3. If stat unitgraph says most of the time is being spent on the GPU thread, a more detailed breakdown can be obtained by running the Unreal GPU profiler. The command is ProfileGPU in the console. A screenshot of that could also be very helpful to diagnose the problem.

https://docs.unrealengine.com/en-US/Engine/Performance/GPU/index.html

@yannbouteiller
Copy link
Author

yannbouteiller commented Nov 4, 2019

Hello @saihv , to open the unreal console, do you mean press ~ when the mouse is captured in the airsim window? Because nothing happens when I do this. (I have a azerty keyboard but even when I switch to qwerty and press ` , or shift + ` , which is supposed to give ~, nothing happens)

@saihv
Copy link
Contributor

saihv commented Nov 4, 2019

Interesting - yes, all you would have to do is focus on the window and then press `. (My bad, it's just the back-tic and not the tilde itself, although Unreal refers to it as Tilde, so you shouldn't need shift). Could you make sure the keypress is actually sending that character through, say, notepad?

Just FYI, the console is super tiny, and usually shows up as a little textbox on the bottom of the screen.

@yannbouteiller
Copy link
Author

yannbouteiller commented Nov 4, 2019

Sorry my bad, I was messing up with the french keyboard. The screenshots for unitgraph are:

EDIT: the screenshots are not representative, I'll make them again.

@yannbouteiller
Copy link
Author

Ok so the screenshots are:

240*320 images no framerate locking:

Screenshot from 2019-11-04 15-32-00

24*32 images no framerate locking:

Screenshot from 2019-11-04 15-37-00

stopped capturing images no framerate locking:

Screenshot from 2019-11-04 15-40-28

Setting t.maxFPS doesn't seem to change anything.

@saihv
Copy link
Contributor

saihv commented Nov 4, 2019

Okay, upon first looks, I think there's something wrong here. Your game frame rate is still decently high, and what I find curious is that your draw thread is taking almost as long as the GPU (rendering) itself (IIRC, Draw is the time it takes to prepare the draw calls on the CPU, GPU is how long it actually takes to render them) which makes me think the GPU is being left waiting for something. For instance, this is a screenshot from my PC while capturing images from one drone.

image

Could you please try the 24x32 image size with our native benchmarking script - with --level_name Soccer_Field_Easy as the argument.

@ironclownfish Any thoughts?

@yannbouteiller
Copy link
Author

Sorry again, t.maxFPS does change something.

320*240 images with 30 game FPS:

Screenshot from 2019-11-04 16-01-44

320*240 images with 20 game FPS:

Screenshot from 2019-11-04 16-22-34

320*240 images with 15 game FPS:

Screenshot from 2019-11-04 16-23-22

320*240 images with 5 game FPS:

Screenshot from 2019-11-04 16-24-29

I don't really know how to interpret these graphs

@saihv
Copy link
Contributor

saihv commented Nov 4, 2019

@yannbouteiller t.maxFPS is supposed to lock the main game framerate to a certain number, which it is doing as you can see by the 'Frame' time on the right. I was mainly curious about whether the image capture FPS is increasing for you when you lock the FPS of the full game.

@yannbouteiller
Copy link
Author

yannbouteiller commented Nov 4, 2019

It possibly improves the general frame rate a tiny bit but setting it to 15 I am still around 7 FPS for both 240 * 320 and 24 * 32 images (retrieved on both drones), all the time being spent waiting in tornado's IOLoop.

I will try your benchmarking script.

@yannbouteiller
Copy link
Author

Using the native benchmarking script I get similar results: for 1 drone, about ~20 FPS for 240 * 320 images and ~ 23 FPS for 24 * 32 images.

@madratman
Copy link
Contributor

@yannbouteiller, can you try the following:

  • In your settings.json, add "ViewMode": "NoDisplay" - or just use airsim_neurips_god_settings_nodisplay.json

  • In the console, first type r.Vsync 0 to assert frame rate is not limited with monitor refresh speed, then t.maxFPS 10000 (some big number to see what's the max rate you can get from your machine), and then run the benchmarking script and tell us what you get. You can ofcourse run Stat FPS and Stat UnitGraph to see if unreal frame rate increases.

@yannbouteiller
Copy link
Author

yannbouteiller commented Nov 6, 2019

Then the image FPS becomes much better, however there is still not a huge gain in performance from 240 * 320 to 24 * 32 :

240 * 320:

Screenshot from 2019-11-05 22-59-47

24 * 32:

Screenshot from 2019-11-05 23-05-02

Is there a way to enforce these unreal options without using the unreal console through the visualization window ? This sounds pretty much exactly like what I want to do for tier 3 in my computing cluster in the other issue (#112)

@madratman
Copy link
Contributor

madratman commented Nov 11, 2019

Even though I have been able to find ways to do this, I am not really able to do this :\ . See below

This can be done via ini files. In AirSimExe/Saved/Config/LinuxNoEditor, make a ConsoleVariables.ini and dump

[Startup]
r.VSync=0
t.MaxFPS=10000

in it.
Source: https://forums.dovetailgames.com/threads/a-guide-to-engine-ini-settings.4672/

Instead of ConsoleVariables.ini, I have tried adding it to Engine.ini and GameUserSettings.ini to no avail.

Next in line are the unreal binary command line options. "Rendering" section is relevant. https://docs.unrealengine.com/en-US/Programming/Basics/CommandLineArguments/index.html
I have been able to disable vsync via the NoVSync option (./AirSimExe.sh -windowed -opengl4 -NoVSync but for the life of me, I can't set MaxFPS.
I did try the FPS command listed therein to no avail.

@saihv @ironclownfish if you have any insight, please comment

@madratman
Copy link
Contributor

madratman commented Nov 11, 2019

@yannbouteiller I just gave the benchmark option a shot, seems to be it:

./AirSimExe.sh -windowed -NoVSync -BENCHMARK

I am not quite sure how it will affect everything else
BENCHMARK: Run game at fixed-step in order to process each frame without skipping any frames. This is useful in conjunction with DUMPMOVIE options.

After some testing, everything seems to be working well. This is what you need, indeed.

madratman added a commit to madratman/AirSim-NeurIPS2019-Drone-Racing that referenced this issue Nov 23, 2019
madratman added a commit to madratman/AirSim-NeurIPS2019-Drone-Racing that referenced this issue Nov 23, 2019
madratman added a commit to madratman/AirSim-NeurIPS2019-Drone-Racing that referenced this issue Nov 23, 2019
madratman added a commit to madratman/AirSim-NeurIPS2019-Drone-Racing that referenced this issue Apr 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants