How to resume training from the last checkpoint?

[Aloento]

I'm now training a model but it's interrupted. So I want to know how to resume training from the last checkpoint?

[erogol]

use python TTS/bin/train_tts.py --continue_path flag

[skol101]

When one resumes, the training still starts from 0 to N (which is 1000 by default). So, do I have to lower the 'epocs' value before resuming?

[jclouet]

To add to this, you can use a pattern as follows to continue multiple device training:

CUDA_VISIBLE_DEVICES="0, 1, 2," python TTS/bin/distribute.py --script path/to/train.py  python TTS/bin/train_tts.py --continue_path path/to/training/model/ouput/checkpoint/

By default, it looks like all the checkpoint directories in the model output file are prefix coqui_tts with a timestamp in the name.