[RLlib] Add separate learning rates for policy and alpha to SAC.

[simonsays1980]

Why are these changes needed?

It is commong practice to use a smaller learning rate for the policy than for the critic in SAC to

• Ensure that the value function gives good approximations for policy improvement
• To keep both learning processes at a similar pace (usually the critic can take larger values than the policy)

This PR proposes separate learning rates for policy, critic and alpha (the hyperparameter to guide entropy regularization over time). It does so by introducing two more arguments to the SACConfig.training method, namely:

• policy_lr
• alpha_lr

Furthermore, it adapts these learning rates in the tuned examples for SAC.

Note, this also enables different learning rates for CQL, which directly inherits from SAC. This should improve learning for both algorithms, SAC and CQL.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[sven1977]

Suggestions:

1. Let's add a third property self.critic_lr and set self.lr to None by default. This increases clarity and expressiveness. Otherwise, users (and us!) will forever have to open this sac.py file, just to quickly check, which one of the three lrs is the one covered by the default self.lr, and which 2 have their own property.
2. Add to validate() a quick check for a) new stack and - if yes - b) self.lr must be None, otherwise raise an informative error explaining that there are 3 different learning rates properties and self.lr should NOT be used.
3. Switch config.lr vs config.critic_lr in the respective SACLearner methods.

[sven1977]

LGTM now! Thanks @simonsays1980 :)