[RLlib] Fix stateful module errors with inference only mode.

[simonsays1980]

Why are these changes needed?

Stateful models need to collect states from the critic during sampling, therefore they cannot be inference-only. This PR fixes this error by setting inference-only=False for stateful modules and checking statefulness in the get_state.

Related issue number

Related to #44758

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[sven1977]

Shouldn't we also use is_stateful() here? What if the user doesn't use the built-in use_lstm option, but comes with their own stateful model?

[simonsays1980]

@sven1977 this was my first intend, however at this point in time is_stateful() cannot be called, yet b/c the encoder is not yet defined.

[simonsays1980]

I agree that this is not a nice solution, but at this point in the code we need to know, if the module is stateful or not, but the is_stateful() depends on the encoder which is defined depending on inference-only being True/False.

[sven1977]

Just one question about use_lstm not being generic enough as a criterion to determine statefulnes..

[simonsays1980]

Just one question about use_lstm not being generic enough as a criterion to determine statefulnes..

I replaced this now with a more generic approach using RecurrentEncoderConfig (we might need to define in the future a StatefulEncoderConfig when attention is joining the club).