Adjust the TuneCallback class for Pytorch-lightning v1.6+ (removing o… #27770
Conversation
…r modifying deprecated functions), reorganized the order to match the rest of the functions, along with the integration tests. Signed-off-by: Davina Zaman <[email protected]>
| def on_epoch_start(self, trainer: Trainer, pl_module: LightningModule): | ||
| if "epoch_start" in self._on: | ||
| def on_train_epoch_start(self, trainer: Trainer, pl_module: LightningModule): | ||
| if "train_epoch_start" in self._on: | ||
| self._handle(trainer, pl_module) |
There was a problem hiding this comment.
@richardliaw should we ensure backwards compatible support for PTL < 1.6?
If so, we can do some versioning checks of pytorch_lightning similar to https://github.com/ray-project/ray/pull/27395/files
There was a problem hiding this comment.
So I looked into implementing this. My proposed solution would be to have an additional class, something like TuneCallbackBackwardCompat that is the old class, and then we inherit from either TuneCallback or the backward compatible version in the remaining classes TuneReportCallback, _TuneCheckpointCallback, TuneReportCheckpointCallback.
Additionally in the tests, we'd need to test both, right?
There was a problem hiding this comment.
Yeah we should ensure backwards compat support. Having a backwards compat class seems good to me, thanks!
There was a problem hiding this comment.
@richardliaw Awesome, I've added those changes in the newest commit
|
Thanks @davzaman for the contribution and @matthewdeng for the initial review! Ping @richardliaw for the question "should we ensure backwards compatible support for PTL < 1.6?" |
…ust tests Signed-off-by: Davina Zaman <[email protected]>
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
|
stale
bot
added
the
stale
|
@richardliaw Does everything look good to go? |
|
Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message. Please feel free to reopen or open a new issue if you'd still like it to be addressed. Again, you can always ask for help on our discussion forum or Ray's public slack channel. Thanks again for opening the issue! |
stale
bot
removed
the
stale
There was a problem hiding this comment.
Sorry for the late review. Generally looks good to me. cc @amogkam for another pass!
| from ray import tune | ||
| from ray.tune.integration.pytorch_lightning import ( | ||
| TuneReportCallback, | ||
| TuneReportCheckpointCallback, | ||
| _TuneCheckpointCallback, | ||
| ) | ||
|
|
||
| ray_pl_use_master = v_parse(version("pytorch_lightning")) >= v_parse("1.6") |
There was a problem hiding this comment.
Looks like we may need the same changes in https://github.com/ray-project/ray/blob/master/python/ray/tests/ray_lightning/simple_tune.py
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
| ray_pl_use_master = v_parse(version("pytorch_lightning")) >= v_parse("1.6") |
There was a problem hiding this comment.
nit: Would rename this to something more descriptive since master may change in the future (e.g. use_ptl_1_6_api)
|
Sorry it took me a while to get to this. I'm seeing @amogkam's commit 60cde11 from a PR #30045 that uses a different solution (dynamic rather than using a flag to decide on using one set of hooks over the other). In that case, the changes I made have already been integrated in a sense. The only thing missing is the update to https://github.com/ray-project/ray/blob/master/python/ray/tests/ray_lightning/simple_tune.py as you mentioned, as well as rerunning https://github.com/ray-project/ray/blob/master/doc/source/tune/examples/tune-pytorch-lightning.ipynb to regenerate the outputs without the deprecation warnings. In that case, how would you like to proceed? We can probably close this pr and just add a commit for the remaining couple of things? |
|
Checking in to see your thoughts re: my previous comment @matthewdeng :) |
|
@davzaman yeah that sounds great! Closing this PR and creating a new one with the remaining changes makes sense to me. |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
|
stale
bot
added
the
stale
|
Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message. Please feel free to reopen or open a new issue if you'd still like it to be addressed. Again, you can always ask for help on our discussion forum or Ray's public slack channel. Thanks again for opening the issue! |
|
Hi @davzaman , |
Why are these changes needed?
This is a duplicate of #27766 but with the changes that were supposed to be there (accidentally made changes while the PR was being accepted while trying to change the commits to have the sign off).
Getting a ton of warnings from pytorch-lightning about deprecation of certain hooks. The TuneCallback has old hooks such as keyboard_interrupt which have been deprecated. I updated the callback to the new hooks and updated the integration tests.
The deprecated hooks and their new versions (In pl v1.6+) are:
on_keyboard_interrupt->on_exceptionon_init_start-> just deprecated no new versionon_init_end-> just deprecated no new versionon_batch_start->on_train_batch_starton_batch_end->on_train_batch_endon_epoch_end->on_<train/validation/test>_epoch_endon_epoch_start->on_<train/validation/test>_epoch_startRelated issue number
Closes #27487
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.I spent a few hours trying to wrangle my environment. Unfortunately I'm on windows using WSL and have an old version of Debian for some reason which only supports gcc-6 (not even close to gcc-9 which is required for building the project so I can run the tests). Given this roadblock and the fact that the fixes were minimal and simple I decided to push out the changes.