[RLlib] APPO Training iteration fn.

[sven1977]

APPO Training iteration fn.

Why are these changes needed?

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[avnishn]

I have 1 question but otherwise lgtm

[avnishn]

This is all looking pretty good, although is it learning?

[sven1977]

Hey @avnishn , please give this another go. I confirmed learning Pong is still good, e.g. on only 7 workers:

(base) ray@ip-172-31-83-222:~/riot_games_atari_benchmarks/ray$ tail ~/output.txt 
Resources requested: 0/8 CPUs, 0/1 GPUs, 0.0/35.56 GiB heap, 0.0/17.78 GiB objects (0.0/1.0 accelerator_type:V100)
Result logdir: /home/ray/ray_results/pong-appo
Number of trials: 1/1 (1 TERMINATED)
+-------------------------------------+------------+---------------------+--------+------------------+---------+----------+----------------------+----------------------+--------------------+
| Trial name                          | status     | loc                 |   iter |   total time (s) |      ts |   reward |   episode_reward_max |   episode_reward_min |   episode_len_mean |
|-------------------------------------+------------+---------------------+--------+------------------+---------+----------+----------------------+----------------------+--------------------|
| APPO_PongNoFrameskip-v4_7fb9b_00000 | TERMINATED | 172.31.83.222:15223 |    154 |          1697.41 | 2931200 |    18.07 |                   20 |                    8 |            7679.47 |
+-------------------------------------+------------+---------------------+--------+------------------+---------+----------+----------------------+----------------------+--------------------+