Add example of saving / resuming DataLoader state with PyTorch Lightning

[schopra8]

📚 Documentation

Right now, the README shows an example of pausing and resuming the StreamingDataLoader state within a simple for loop over the dataloader.

It's not clear to me how to adapt that example to work with PyTorch Lightning, where the training loop is abstracted away.

My first guess is to make a callback for the Trainer -- but it would be great to have a simple example of how to do this in PyTorch Lightning.

[awaelchli]

The README shows how you'd do it in plain torch yes. PyTorch Lightning already supports checkpointing out of the box, so you'd literally have to do nothing.

If your train_dataloader() methods return a StreamingDataLoader then its state will be saved into the checkpoints and when you resume from one via trainer.fit(..., ckpt_path=...) it loads the dataloader state back to continue.

Yes we can make it more explicit in docs and examples etc, good idea!

[schopra8]

Thank you for the clarification!