Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RLlib] Add 2D box example for PPO RL Modules #33840

Merged
merged 62 commits into from
Apr 14, 2023

Conversation

ArturNiederfahrenhorst
Copy link
Contributor

Why are these changes needed?

Historically, RLlib has simply flattened 2D inputs.
This often makes it hard for algorithms to learn in such spaces.
The Catalog introduced with RLModules throws an error when facing such spaces instead of silently flattening them and tells users to transform observations into a 3D space.
This example shows how this can be done on a pettingzoo env.

kouroshHakha and others added 30 commits February 23, 2023 22:08
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
…e minibatch, it was the entire batch.

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
… that was just added

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
… SingleAgentRLModule

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
@ArturNiederfahrenhorst
Copy link
Contributor Author

ArturNiederfahrenhorst commented Mar 29, 2023

Executing the example leads to the following learning curve:

Screenshot 2023-03-28 at 23 31 52

https://tensorboard.dev/experiment/xlxnhV0jQBqK13iRJvRzTA/#scalars

Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Signed-off-by: Artur Niederfahrenhorst <[email protected]>
@ArturNiederfahrenhorst ArturNiederfahrenhorst changed the title [RLlib] Add 2d box example [RLlib] Add 2D box example for PPO RL Modules Mar 29, 2023
Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Copy link
Contributor

@kouroshHakha kouroshHakha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this not the same as what we have in the unittest?

@ArturNiederfahrenhorst
Copy link
Contributor Author

@kouroshHakha Yes, we have unittests that cover these inputs.
But we don't have an example that shows how, after RLModules, users should interface with environments that have a 2D obs space. The solution here is simply to wrap the environment and transform the observation space into something that RLlib likes. This is a simple solution that lets users write python code to solve their problem instead of implicitly transforming their inputs which has caused a lot of confusion in the past.

@gjoliver gjoliver merged commit a333017 into ray-project:master Apr 14, 2023
vitsai pushed a commit to vitsai/ray that referenced this pull request Apr 17, 2023
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: elliottower <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants