[RLlib] Fix default view_requirement in policy.py

[kouroshHakha]

Why are these changes needed?

the default view_requirement in policy.py used to enable used_for_compute_actions for new_obs key; for inference we should not look into the future.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[gjoliver]

looks good. should we mark INFOS column not_for_compute_action as well?
last time I had trouble with this, I made some custom changes in RandomPolicy:
https://github.com/ray-project/ray/blob/master/rllib/examples/policy/random_policy.py#L36-L40
if we flip the default values here, then we can get rid of the custom fix.
I don't think info dict will ever be able to get batched and used for forward pass directly?

[kouroshHakha]

looks good. should we mark INFOS column not_for_compute_action as well? last time I had trouble with this, I made some custom changes in RandomPolicy: https://github.com/ray-project/ray/blob/master/rllib/examples/policy/random_policy.py#L36-L40 if we flip the default values here, then we can get rid of the custom fix. I don't think info dict will ever be able to get batched and used for forward pass directly?

None of the other default values should hit this particular issue, since they are not forward looking views?

[gjoliver]

looks good. should we mark INFOS column not_for_compute_action as well? last time I had trouble with this, I made some custom changes in RandomPolicy: https://github.com/ray-project/ray/blob/master/rllib/examples/policy/random_policy.py#L36-L40 if we flip the default values here, then we can get rid of the custom fix. I don't think info dict will ever be able to get batched and used for forward pass directly?

None of the other default values should hit this particular issue, since they are not forward looking views?

right, they should not hit the particular issue.
it's just the dict values that should supposedly break our batch building logic.