[RLlib]: Cleanup examples folder: Add example restoring 1 of n agents from a checkpoint. #45462
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
… multi-agent environment. Signed-off-by: Simon Zehnder <[email protected]>
simonsays1980
added
rllib
Signed-off-by: Simon Zehnder <[email protected]>
sven1977
changed the title
sven1977
requested review from
sven1977,
avnishn,
ArturNiederfahrenhorst,
maxpumperla and
kouroshHakha
as code owners
sven1977
reviewed
May 21, 2024
sven1977
reviewed
May 21, 2024
rllib/examples/multi_agent/restore_1_of_n_agents_from_checkpoint.py
Outdated
Show resolved
Hide resolved
sven1977
reviewed
May 21, 2024
rllib/examples/multi_agent/restore_1_of_n_agents_from_checkpoint.py
Outdated
Show resolved
Hide resolved
sven1977
reviewed
May 21, 2024
| @@ -0,0 +1,151 @@ | |||
| """Simple example of loading module weights for 1 of n agents from checkpoint. | |||
There was a problem hiding this comment.
nit: Let's not use "simple".
"An example script showing how to load RLModule weights for 1 out of n agents from a checkpoint" ?
There was a problem hiding this comment.
Yup, not simple for everyone lol. I know what you mean, its actually quite some complexity to make this possible in MA scenarios - and so powerful.
sven1977
reviewed
May 21, 2024
| @@ -0,0 +1,151 @@ | |||
| """Simple example of loading module weights for 1 of n agents from checkpoint. | |||
|
|
|||
There was a problem hiding this comment.
Can we add a tiny paragraph here saying the usual:
This example:
- runs a multi-agent Pendulum experiment with ... policies ... blabla
- saves a checkpoint of the used MultiAgentRLModule every blabla iterations
- stops the experiment after the agents reach a combined return of ...
- picks the best of both trained policies (based on episode return) and restores only the corresponding RLModule.
- runs a second experiment with the restored RLModule (single-agent) .... blabla
sven1977
approved these changes
May 21, 2024
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
…he issues. In addition added 'no_main' tag to test in BUILD b/c linter errored out. Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
can-anyscale
added a commit
that referenced
this pull request
May 22, 2024
#45462 adds a new tests by changing bazel rule instead of adding a new test file; this case can only be covered by our previous logic of computing new tests; recover this logic (in addition to the logic of computing new tests by looking at changed test files) Test: - CI --------- Signed-off-by: can <[email protected]>
can-anyscale
added a commit
that referenced
this pull request
May 23, 2024
#45462 adds a new tests by changing bazel rule instead of adding a new test file; this case can only be covered by our previous logic of computing new tests; recover this logic (in addition to the logic of computing new tests by looking at changed test files) This is a redo of #45495 which got reverted. The difference now is that we run the bazel command in a container instead of on the current environment. bazel seems to have issues sharing the cache when calling bazel within bazel (https://buildkite.com/ray-project/microcheck/builds/444#018fa23a-6e31-435b-a0ea-412ca2d1017b/175-1476) Test: - CI - full microcheck run: https://buildkite.com/ray-project/microcheck/builds/464 Signed-off-by: can <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Signed-off-by: Simon Zehnder <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Restoring certain agents from checkpoint is a frequent use case and we should provide examples for this scenario. This PR is adding such an example in the new API. stack. The example does the following:
nagents onPendulum-v1MultiEnv.policy 0from this checkpoint.policy 0restored from checkpoint.This example shows that training further on from a restored checkpoint - even for only a single agent - results in faster convergence.
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.