[RLlib] Added benchmark experiment for SAC with MuJoCo, PPO with MuJoCo and DQN with Atari.

[simonsays1980]

Why are these changes needed?

This PR adds some benchmark runs for the following algorithms:

1. DQN: Atari & Atari with PB2
2. SAC: MuJoCo & MuJoCo with PB2 (HalfCheetah-v4, Hopper-v4, Humanoid-v4, Ant-v4, Walker2d-v4)
3. PPO: MuJoCo & MuJoCo with PB2

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[sven1977]

Awesome PR @simonsays1980 ! Did we actually run these on RLlib and got these results? Or how did you get to these episode-return numbers?

[simonsays1980]

Thanks ;) No we did not - I collected results from the papers. Take a look at my slack comment towards it, too.

[sven1977]

Had no idea "env" is always part of the results dict. ?

[simonsays1980]

I think it is only when it is chosen by tunegird_search`

[sven1977]

These are MuJoCo bechmarks. Are there respective results reported for CleanRL for Atari?

[simonsays1980]

Yes there are - but not for all of the Atari games I think.

[sven1977]

Could we do a quick comparison of this wrapper (including the settings we use below, dim=84, noop=0, grayscale=True) vs our own Atari wrapper func in rllib.env.wrappers.atari_wrappers::wrap_atari_for_new_api_stack()?

RLlib nowadays (new API stack) uses a DreamerV3 style Atari setup:
dim=64
grayscale=True
noop=30
action_repeat_prob=0.0 (<- deterministic!)
small action space=True

I'd be totally happy with replacing our function with this gymnasium wrapper here, but I want to make sure we don't mess up our existing benchmarks.

[simonsays1980]

We have to keep in mind that I took the benchmarks from the DQN Rainbow paper that is already some years old - there they use these settings. DreamerV3 is newer and therefore has taken - maybe - advantage of some developments there.

The ultimate question is: to what do we want to compare our algorithms?

[sven1977]

I think we should just be bullish and:
a) Use our well-proven hyperparams and network architectures (e.g. the DreamerV3 CNN stack mentioned above, which has proven to be strong on Atari tasks with e.g. PPO).
b) The reason users use RLlib is not b/c it runs 20% faster or 2% slower than SB3 or another library. It's b/c it's scalable, multi-agent capable, multi-GPU capable, etc.. So as long as we show proper benchmark results (with one set of hyperparams or another) and maybe how scaling (adding more workers) affects these results, we should be solid.

[sven1977]

c) If we do this, then we won't have the problem of always having to "look back" and backward-support these quite old settings (DQN paper 2015 -> almost 10 years old). We can freely find new and better parameters and scaling settings as long as they improve the performance.

[simonsays1980]

I agree on your arguments. I added the PB2 run files to find such parameters and perform as best as possible with our setup and as discussed - we should run it with 1, 2, 4, x GPUs to show scalability.

[simonsays1980]

In regard to the atari preprocessing. Let's make two single runs - one with our preprocessing and another with the gymnasium one and see which one performs better. It suffices to run for maybe 100-200 iterations imo.

[simonsays1980]

@sven1977 Tbh I cannot find the settings for DreamerV3 using this wrap_atari_for_new_api_stack. The function also has no grayscale to be set. It uses a specific env_config:

# [2]: "We follow the evaluation protocol of Machado et al. (2018) with 200M
# environment steps, action repeat of 4, a time limit of 108,000 steps per
# episode that correspond to 30 minutes of game play, no access to life
# information, full action space, and sticky actions. Because the world model
# integrates information over time, DreamerV2 does not use frame stacking.
# The experiments use a single-task setup where a separate agent is trained
# for each game. Moreover, each agent uses only a single environment instance.
env_config={
    # "sticky actions" but not according to Danijar's 100k configs.
    "repeat_action_probability": 0.0,
    # "full action space" but not according to Danijar's 100k configs.
    "full_action_space": False,
    # Already done by MaxAndSkip wrapper: "action repeat" == 4.
    "frameskip": 1,
}

[sven1977]

Should this be False?

[simonsays1980]

Shouldn't PPO - having a stochastic policy - explore in evaluation to keep performance? I remember such discussions on the forum about the old stack PPO where users had this set to False and encountered a drop in performance.

[sven1977]

Just a few nits before we can merge:

• question on explore=True/False in eval config
• use our own Atari wrapper in the Atari benchmarks

[sven1977]

Approved! Thanks @simonsays1980 . Can be merged pending comments/nits.