[RLlib] QMIX better defaults + added to CI learning tests

[sven1977]

• Add QMIX learning tests (two_step_game) to CI for all available "mixers" (qmix, vdn, None).
• Change QMIX default exploration config from final_epsilon=0.02 to 0.01 and epsilon_timesteps=10k to 40k. The new values are the RLlib original ones (had been changed with the introduction of the exploration API, when we switched to the default values of the EpsilonExploration class).
• Made QMIX unit test more complex (2 agents grouped instead of just 1).

Why are these changes needed?

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[gjoliver]

cool. so the regression is just caused by hyper parameters?

[gjoliver]

try

frameworks: ["torch"] 

at the top level to only run these tests for torch?
I see that these new tests are failing because of framework tf.