[RLlib] Fix adding policies to rolloutworkers with complex and discrete observation spaces

[ArturNiederfahrenhorst]

Signed-off-by: Artur Niederfahrenhorst [email protected]

Why are these changes needed?

Filtering complex spaces with a mixture of discrete/multidiscrete and continuous spaces won't work if you add a new policy to a rollout worker. This is because filters expect a space containing Nones instead of such discrete spaces.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[sven1977]

Nice catch. Thanks for the fix Artur.