[RLlib] Torch algos use now-framework-agnostic MultiGPUTrainOneStep execution op (~33% speedup for PPO-torch + GPU).

[sven1977]

This PR makes all torch algos use the formerly TrainTFMultiGPU (now made framework agnostic and renamed into MultiGPUTrainOneStep).
This allows for pre-loading of batches into GPU memory and thus not having to re-load them every time we do an SGD iter.

First experiments suggest a ~33% speedup on a single GPU for PPO torch on Atari and a ~60+% speedup for 2GPUs)!

Benchmarks for measuring PPO torch speedups (1 GPU + 11 workers on Breakout-v4 on a p3dn.24xlarge machine; batch size 5k; lr=0.00005):

This PR:
== Status ==
Memory usage on this node: 17.4/747.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 11.0/96 CPUs, 1.0/8 GPUs, 0.0/550.03 GiB heap, 0.0/186.26 GiB objects (0.0/1.0 accelerator_type:V100)
Result logdir: /home/ray/ray_results/atari-ppo
Number of trials: 1/1 (1 RUNNING)
+----------------------------------------+----------+-------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+
| Trial name                             | status   | loc               |   iter |   total time (s) |     ts |   reward |   episode_reward_max |   episode_reward_min |   episode_len_mean |
|----------------------------------------+----------+-------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------|
| PPO_BreakoutNoFrameskip-v4_47de3_00000 | RUNNING  | 172.31.80.77:1289 |     51 |          210.334 | 255000 |      1.1 |                    5 |                    0 |             723.76 |
+----------------------------------------+----------+-------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+


Ray 1.4
== Status ==
Memory usage on this node: 17.4/747.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 11.0/96 CPUs, 1.0/8 GPUs, 0.0/549.92 GiB heap, 0.0/186.26 GiB objects (0.0/1.0 accelerator_type:V100)
Result logdir: /home/ray/ray_results/atari-ppo
Number of trials: 1/1 (1 RUNNING)
+----------------------------------------+----------+-------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+
| Trial name                             | status   | loc               |   iter |   total time (s) |     ts |   reward |   episode_reward_max |   episode_reward_min |   episode_len_mean |
|----------------------------------------+----------+-------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------|
| PPO_BreakoutNoFrameskip-v4_17f4f_00000 | RUNNING  | 172.31.80.77:8721 |     51 |          276.744 | 255000 |     3.49 |                   14 |                    0 |             999.73 |
+----------------------------------------+----------+-------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+

For 2 GPUs, the situation changed as follows (doubling batch size to 10k and lr to 0.0001):

This PR:

== Status ==
Memory usage on this node: 18.9/747.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 11.0/96 CPUs, 2.0/8 GPUs, 0.0/549.94 GiB heap, 0.0/186.26 GiB objects (0.0/1.0 accelerator_type:V100)
Result logdir: /home/ray/ray_results/atari-ppo
Number of trials: 1/1 (1 RUNNING)
+----------------------------------------+----------+--------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+
| Trial name                             | status   | loc                |   iter |   total time (s) |     ts |   reward |   episode_reward_max |   episode_reward_min |   episode_len_mean |
|----------------------------------------+----------+--------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------|
| PPO_BreakoutNoFrameskip-v4_f3219_00000 | RUNNING  | 172.31.80.77:90505 |     51 |          318.461 | 510000 |     9.53 |                   21 |                    3 |            1703.92 |
+----------------------------------------+----------+--------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+


Ray 1.4:

== Status ==
Memory usage on this node: 18.8/747.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 11.0/96 CPUs, 2.0/8 GPUs, 0.0/549.89 GiB heap, 0.0/186.26 GiB objects (0.0/1.0 accelerator_type:V100)
Result logdir: /home/ray/ray_results/atari-ppo
Number of trials: 1/1 (1 RUNNING)
+----------------------------------------+----------+--------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+
| Trial name                             | status   | loc                |   iter |   total time (s) |     ts |   reward |   episode_reward_max |   episode_reward_min |   episode_len_mean |
|----------------------------------------+----------+--------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------|
| PPO_BreakoutNoFrameskip-v4_ff193_00000 | RUNNING  | 172.31.80.77:26690 |     51 |          502.618 | 510000 |     8.35 |                   17 |                    2 |            1565.59 |
+----------------------------------------+----------+--------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+

Why are these changes needed?

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(