Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] DreamerV3: Catalog enhancements 05 - GRU default model support #34284

Merged
merged 59 commits into from
Apr 18, 2023
Merged

[RLlib] DreamerV3: Catalog enhancements 05 - GRU default model support #34284

merged 59 commits into from
Apr 18, 2023

Conversation

sven1977
Copy link
Contributor

@sven1977 sven1977 commented Apr 11, 2023 •
edited
Loading

DreamerV3: Catalog enhancements 05 - GRU default model support

  • Generalize LSTM- and GRU-based encoders into one single config (RecurrentEncoderConfig) for simplicity.
  • Both can be layer-stacked with state=(num_layers, B, [hidden state dim]).
  • GRU encoders have the same architecture and compute-behavior between tf2 and torch (confirmed with test cases).

Why are these changes needed?

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

sven1977 and others added 30 commits March 28, 2023 16:42
@sven1977
wip
36a4c65 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
fd432e1 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into drea…
eb33ba0 
…mer_v3_catalog_enhancements_01
@sven1977
wip
36346cf 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into drea…
5995906 
…mer_v3_catalog_enhancements_01
@sven1977
wip
950eb81 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
392b5f6 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
2cce48c 
Signed-off-by: sven1977 <[email protected]>
@sven1977
fix
46b3c4f 
Signed-off-by: sven1977 <[email protected]>
@sven1977
removed torch SiLU (torch now has its own)
1ab97e7 
Signed-off-by: sven1977 <[email protected]>
@sven1977
LINT
6383061 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into drea…
1413b98 
…mer_v3_catalog_enhancements_01
@sven1977 @kouroshHakha
Update rllib/core/models/base.py
dc46838 
Co-authored-by: kourosh hakhamaneshi <[email protected]>
Signed-off-by: Sven Mika <[email protected]>
@sven1977
wip
9427f9b 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
5f91341 
Signed-off-by: sven1977 <[email protected]>
@sven1977
LINT and cleanup
647545d 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into drea…
6e88ecc 
…mer_v3_catalog_enhancements_01
@sven1977
Merge branch 'model_spec_rename_to_specs_plural_to_avoid_tf_keras_mod…
14a277b 
…el_conflict' into dreamer_v3_catalog_enhancements_01

Signed-off-by: sven1977 <[email protected]>

# Conflicts:
#	rllib/BUILD
#	rllib/core/models/base.py
#	rllib/core/models/specs/tests/test_tensor_spec.py
#	rllib/core/models/tf/encoder.py
#	rllib/core/models/tf/mlp.py
#	rllib/core/models/tf/primitives.py
#	rllib/core/models/torch/mlp.py
#	rllib/models/utils.py
@sven1977
Merge remote-tracking branch 'origin/dreamer_v3_catalog_enhancements_…
fe955f6 
…01' into dreamer_v3_catalog_enhancements_01
@sven1977
wip
ebadcfe 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into mode…
fb183ea 
…l_spec_rename_to_specs_plural_to_avoid_tf_keras_model_conflict
@sven1977
test case fix
d802617 
Signed-off-by: sven1977 <[email protected]>
@sven1977
cleanup
4efa147 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'model_spec_rename_to_specs_plural_to_avoid_tf_keras_mod…
3885cba 
…el_conflict' into dreamer_v3_catalog_enhancements_01
@sven1977
wip
6a6370c 
Signed-off-by: sven1977 <[email protected]>
@sven1977
merge
3de4cd1 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
28d0c5e 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
a9c2897 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
16101b8 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into drea…
ed65960 
…mer_v3_catalog_enhancements_02_deconv2d

Signed-off-by: sven1977 <[email protected]>

# Conflicts:
#	rllib/BUILD
#	rllib/core/models/specs/tests/test_tensor_spec.py
#	rllib/core/models/tf/encoder.py
#	rllib/core/models/tf/mlp.py
#	rllib/core/models/tf/primitives.py
#	rllib/core/models/torch/mlp.py
#	rllib/models/utils.py
@sven1977
Merge branch 'dreamer_v3_catalog_enhancements_01' into dreamer_v3_cat…
2cdd41c 
…alog_enhancements_02_deconv2d
@sven1977
merge
80c5a99 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into drea…
4056ab6 
…mer_v3_catalog_enhancements_02_deconv2d
@sven1977
wip
e74db3b 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into drea…
5891668 
…mer_v3_catalog_enhancements_04_lstm
@sven1977
wip
5467f04 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into drea…
883993c 
…mer_v3_catalog_enhancements_05_gru
@sven1977
wip
33b998e 
Signed-off-by: sven1977 <[email protected]>
@sven1977 sven1977 added the do-not-merge Do not merge this PR! label Apr 11, 2023
@sven1977
merge
d2e470b 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
9f783ea 
Signed-off-by: sven1977 <[email protected]>
@sven1977 sven1977 removed the do-not-merge Do not merge this PR! label Apr 16, 2023
@sven1977 sven1977 changed the title [WIP; RLlib] DreamerV3: Catalog enhancements 05 - GRU default model support [RLlib] DreamerV3: Catalog enhancements 05 - GRU default model support Apr 17, 2023
ArturNiederfahrenhorst
ArturNiederfahrenhorst approved these changes Apr 17, 2023
View reviewed changes
Copy link
Contributor

@ArturNiederfahrenhorst ArturNiederfahrenhorst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR looks super cool.
test_recurrent_encoders is currently failing.
So after that, I say we can merge 👍

@sven1977
fixes
2989da3 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Copy link
Contributor Author

Thanks for the review @ArturNiederfahrenhorst , fixed the test and waiting for all to pass now ...

@sven1977 sven1977 merged commit 512b319 into ray-project:master Apr 18, 2023
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@sven1977 @elliottower
[RLlib] DreamerV3: Catalog enhancements 05 - GRU default model suppor…
e6db435 
…t. (ray-project#34284)

Signed-off-by: elliottower <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@sven1977 @ProjectsByJackHe
[RLlib] DreamerV3: Catalog enhancements 05 - GRU default model suppor…
f1307cf 
…t. (ray-project#34284)

Signed-off-by: Jack He <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ArturNiederfahrenhorst ArturNiederfahrenhorst ArturNiederfahrenhorst approved these changes

@gjoliver gjoliver Awaiting requested review from gjoliver

@avnishn avnishn Awaiting requested review from avnishn

@smorad smorad Awaiting requested review from smorad

@maxpumperla maxpumperla Awaiting requested review from maxpumperla

@kouroshHakha kouroshHakha Awaiting requested review from kouroshHakha

@krfricke krfricke Awaiting requested review from krfricke

Assignees

@ArturNiederfahrenhorst ArturNiederfahrenhorst

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sven1977 @ArturNiederfahrenhorst