[RLlib] disabled torch release test on APPO

[kouroshHakha]

Signed-off-by: Kourosh Hakhamaneshi [email protected]

Why are these changes needed?

This test just never passed. For some reason torch version has always been broken. We should disable this for now until this gets fixed. otherwise the release test notifs will be noisy.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[sven1977]

The reason why torch does not learn this task for this test is simply speed (very similar issue with the corresponding PPO test). After a deeper investigation into our PPO+GPU+Atari+CUDA11.6 tests, @smorad and I have found the problem to be simply the .backward() call on the loss. We made sure it's not the GPU->CPU copying, not the optimizer.zero_grad, not the input dtypes (e.g. double instead of float), not the loss math itself, not the model (same number of trainable params than its tf counterpart). We also checked the actual torch graph with the graphviz + torchviz package and saw nothing suspicious. We might have to go back and try different torch+CUDA versions again (we did this once figuring out that CUDA 10.2 was quite good for torch, actually) to overcome these test failures. :/

[kouroshHakha]

let's hold off on this PR until we come to a conclusion with @sven1977 's investigation on torch + cuda thing.

[kouroshHakha]

closing in favor of #30361