Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[rllib] Should remove vf_clip param from PPO #8908

Closed
ericl opened this issue Jun 11, 2020 · 1 comment
Closed

[rllib] Should remove vf_clip param from PPO #8908

ericl opened this issue Jun 11, 2020 · 1 comment
Labels
enhancement Request for new feature and/or capability P2 Important issue, but not time-critical rllib RLlib related issues

Comments

@ericl
Copy link
Contributor

ericl commented Jun 11, 2020

Describe your feature request

According to https://arxiv.org/pdf/2006.05990.pdf, we should remove VF clipping since it doesn't help. VF clipping has historically been a common cause of user problems in RLlib anyways, so this might be a nice double win.

Note: it seems the paper only evaluates relatively small epoch sizes (at most 4096 steps), whereas many of our examples are tuned for much higher batch sizes (up to 320k steps), which tend to reach higher final rewards. We might want to re-benchmark these to make sure VF clipping still doesn't matter.

cc @eugenevinitsky @sven1977

@ericl ericl added enhancement Request for new feature and/or capability triage Needs triage (eg: priority, bug/not-bug, and owning component) P2 Important issue, but not time-critical rllib and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Jun 11, 2020
@ericl ericl added this to the RLlib Bugs milestone Mar 11, 2021
@ericl ericl removed the rllib label Mar 11, 2021
@richardliaw richardliaw added the rllib RLlib related issues label Oct 5, 2021
@avnishn
Copy link
Member

avnishn commented Feb 7, 2022

I'm going to go ahead and close this for now, as it seems that we're going to stick with vf clipping see #22171

@avnishn avnishn closed this as completed Feb 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Request for new feature and/or capability P2 Important issue, but not time-critical rllib RLlib related issues
Projects
None yet
Development

No branches or pull requests

3 participants