[RLlib] MultiAgentEpisode: Fix/enhance cut() API.
#44677
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
sven1977
requested review from
avnishn,
ArturNiederfahrenhorst,
maxpumperla,
kouroshHakha and
simonsays1980
as code owners
simonsays1980
approved these changes
Apr 11, 2024
| successor._hanging_actions_end = copy.deepcopy(self._hanging_actions_end) | ||
| successor._hanging_rewards_end = self._hanging_rewards_end.copy() | ||
| successor._hanging_extra_model_outputs_end = copy.deepcopy( | ||
| # Copy over the hanging (end) values into the hanging (begin) chaches of the |
There was a problem hiding this comment.
"chaches" -> "caches". And can we leave a note here, why we need the _hanging_actions_begin cache here? Why not writing it into the successor._hanging_actions_end ones?
…i_agent_episode_fix_cut
Signed-off-by: sven1977 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MultiAgentEpisode: Fix/enhance
cut()API.Currently, when cutting a multi-agent episode, we do not properly account for sections in which one (or more) agents are not receiving observations (and are accumulating hanging rewards and one hanging action). These hanging values should be added to a different cache (the "before") cache, rather than the usual cache at the end of the episode. This then helps once we need to concatenate different chunks, e.g. one with a end-cache to the left of another chunk with a begin-cache. The caches have to match (actions) and/or get added (rewards) to yield the correct original MultiAgentEpisode.
NOTE: While this enhancement does help with replay buffers, in which we keep concatenating new chunks (with possible begin caches) to already stored chunks (with possible end caches), it still loses single-agent timesteps in the case where we have a) on-policylearning (no replay buffers) AND b) we
cut()the MAEpisode (in the EnvRunner) at exactly a timestep, in which one or more agents are not receiving observations. These single-agent timesteps are lost and cannot be learned from.A solution for this problem could be (but will have to be discussed and implemented) to also store the most recent observation as a hanging one in the before-cache.
Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.