Fixes some pickling issues in the Cythonized Scheduler #4768
Conversation
Cython encounters a bug when pickling unbound class methods. This has been fixed upstream, but not yet released. Include a link to the issue for reference.
Cython encounters a bug when pickling unbound class methods. This has been fixed upstream, but not yet released. To workaround this issue, create a local wrapper function to pickle in its place that just calls the method.
Cython seems to run into issues pickling the `lambda` in `Scheduler.__init__`. Link to the relevant upstream issue for tracking.
Cython seems to run into issues pickling the `lambda` in `Scheduler.__init__`. Instead of using a `lambda`, workaround this by using `partial`. This pickles without problems bypassing the Cython issue.
jakirkham
force-pushed
the
fix_some_cy_xfails
branch
from
ee0cb1c
to
9317312
Compare
| "teardown": dumps_function(Scheduler.remove_plugin), | ||
| "teardown": dumps_function(remove_plugin), |
There was a problem hiding this comment.
Maybe dumb question but Scheduler.remove_plugin isn't a local scope function. Why does this change anything?
There was a problem hiding this comment.
I believe this is a workaround for cython/cython#2972. An upstream fix in cython (cython/cython#2969) should be included in the cython=0.29.24 release
There was a problem hiding this comment.
Yeah exactly
Pickling of the local remove_plugin function should only include the function and module to import. The code itself won't need to be pickled, which avoids needing to pickle Scheduler.remove_plugin
There was a problem hiding this comment.
Thanks for these fixes @jakirkham! I pushed a tiny commit to add a comment about remove_plugin. Will merge after CI finishes
|
Thanks all! 😄 |
A few of the test failures discovered in PR ( #4764 ) were related to pickling issues. One relating to pickling unbound methods, which Cython has a fix for in a pending release ( cython/cython#2972 ). Another related pickling to
lambdas ( cython/cython#3499 ).Here we workaround these issues by using other techniques. This fixes a couple of test failures on the Cython Scheduler. So are now re-enabled in this PR.
black distributed/flake8 distributed/isort distributed