Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[autoscaler]: Assertion in info string can kill autoscaler #3903

Closed
ls-daniel opened this issue Jan 30, 2019 · 1 comment
Closed

[autoscaler]: Assertion in info string can kill autoscaler #3903

ls-daniel opened this issue Jan 30, 2019 · 1 comment

Comments

@ls-daniel
Copy link
Contributor

ls-daniel commented Jan 30, 2019

In python/ray/autoscaler/autoscaler.py there's an info string that gets logged occasionally.

The line assert used >= 0 is present here. I don't think that assertions should be happening in an informational string.

I can't easily reproduce this but I occasionally have this causing issues (if it blows up during an autoscaler.update call more than a few times, eventually the autoscaler will just bomb itself 'Too many errors, abort').

Can I just remove the assertion?

@ericl
Copy link
Contributor

ericl commented Jan 30, 2019

Yeah, we should set used = max(0, used) instead to guard against this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants