-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve errors raised by ds.groupby() of unsupported key type #21610
Conversation
@ericl I remember when we initially discussed the implementation of The current semantics of bool type is that if it's true, all exceptions are retried, but I think we can change the semantics to be only retrying IOError and subclasses. Alternatively, if we don't want to change the semantics of bool type, we can support list of exceptions and do |
@jjyao If it's not much extra effort, I'd vote for adding support for a list of retryable exceptions. Any complications around having Python objects (classes) in task options? Should be able to serialize them as pickle bytes, store those bytes opaquely in the task spec, and eventually deserialize them for the retrying logic during task execution, right? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, nice!
Will follow-up with a PR cleaning up errors for aggregate() and sort() as well. |
Why are these changes needed?
This raises errors early in the driver instead of in the workers. Btw @jjyao we should probably fix our error retry policy to not retry non-IO errors, is there a way to do that today? Perhaps we can raise a fatal type of error for non-IO errors in our tasks.