Add simple FSDP support to MNIST example LightningModule
#604
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
mnist_module.py
to reference theTrainer
's version ofparameters()
e.g., for FSDP supportparameters()
in this way,Lightning
strategies such as FSDP will not be able to successfully wrap one's model parameters. Even more importantly: if one were to train a model while referencingself.parameters()
and then attempt to re-train the model when referencingself.trainer.model.parameters()
,Lightning
2.0 will (currently) raise an Exception, preventing one from resuming any training with the original checkpoint. That is why I think this change is important for everyone to use as a default.Before submitting
pytest
command?pre-commit run -a
command?Did you have fun?
Make sure you had fun coding 🙃