Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] Add correct terminated and truncated batch sizes on zero-length episodes #46721

Merged
merged 2 commits into from
Sep 25, 2024
+4 −0
Merged

[RLlib] Add correct terminated and truncated batch sizes on zero-length episodes #46721

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1d29c33
Add correct terminated and truncated batch sizes on zero-length episodes
Mark2000 Jul 20, 2024
6d54957
Update add_columns_from_episodes_to_train_batch.py
Mark2000 Jul 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions rllib/connectors/learner/add_columns_from_episodes_to_train_batch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,8 @@ def __call__(
Columns.TERMINATEDS,
items_to_add=(
[False] * (len(sa_episode) - 1) + [sa_episode.is_terminated]
if len(sa_episode) > 0
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably remove any zero-length episodes in the loop here. They cannot be used for learning anyways.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leave this to @sven1977 to decide.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sven1977 could we give this a go? I think this is fine like this. The other option is to remove zero-length epiosdes entirely as they offer nothing to learn from. This could also happen if an agent in a multi-agent scenario received only the initial observation and nothing else.

else []
),
num_items=len(sa_episode),
single_agent_episode=sa_episode,
Expand All @@ -115,6 +117,8 @@ def __call__(
Columns.TRUNCATEDS,
items_to_add=(
[False] * (len(sa_episode) - 1) + [sa_episode.is_truncated]
if len(sa_episode) > 0
else []
),
num_items=len(sa_episode),
single_agent_episode=sa_episode,
Expand Down
Loading