[RLlib] New API stack: (Multi)RLModule overhaul vol 05 (deprecate Specs, SpecDict, TensorSpec).

[sven1977]

New API stack: (Multi)RLModule overhaul vol 05 (deprecate Specs, SpecDict, TensorSpec).
Arguments for this move are:

• Simplicity! The fewer classes users have to learn (or just think(!) they have to learn) to get started with RLlib, the better.
• Only useful for default RLlib modules/model components (users know much better what their models and model-components expect and should not have to worry about learning to correctly tag their components with specs or override a dozen or so methods to achieve this kind of tagging: get_input_specs, get_output_specs, input_specs_inference, output_specs_inference, input_specs_exploration, output_specs_exploration, input_specs_train, output_specs_train).
• Those default RLlib models can all be reduced to very simple input/output structures: [B, ...] for non-RNN model components and [B, T, ...] for RNN-based model components. We don't need specs for this.
• On the RLModule side:
  ** All _forward_inference AND _forward_exploration should either return ACTION_DIST_INPUTS and/or ACTIONS keys. The behavior for RLlib in terms of action sampling in the EnvRunners is already well described in the docs.
  ** _forward_train must always be properly aligned with the algo's Learner's loss function, so this one remains tricky, regardless of having specs or not. The long-term solution should be what we have already started with the RLModule APIs: The algo's Learner determines, which APIs are hard requirements for any RLModule (custom or RLlib-default) to bring to the table in order to be learnt by the algo, for example: PPO requires all RLModules to implement the ValueFunctionAPI. We should expand this algo-agnostic and spec independent pattern to all the other algos as well (DQN/SAC/DreamerV3, etc..).

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[simonsays1980]

LGTM. A lot of code will be gone which means: less maintenance costs - this is good! I have some worries in regard to error identifiability. We might want to test this.

[simonsays1980]

Can we leave these as a kind of comment/docstring in the forward method? Maybe we do this for all default algos such that we, new RLlib team members and users always know what is returned (received).

[simonsays1980]

Let us check with tests, if missing a needed input or output is well traceable and the error can be well understood.