always_save_nemo not working properly

[aimarz]

I'm using nemo 24.07 container to train LLMs.

I want to save checkpoints in .nemo format, so I set exp_manager.checkpoint_callback_params.always_save_nemo: true. And it works in the sense that it saves the checkpoint as a .nemo, but the problem is that all the files get the same name, regardless of the step. Therefore, it constantly replaces the same .nemo file, and, in the end of the training, I just get the last checkpoint. So it is useless.

I have tried setting postfix: {step}.nemo like it is done with the megatron checkpoint, but it does not work. It does not seem like any .format() is called on it, so I don't know how to change the name depending on the step number.

Please fix this issue somehow.

[ericharper]

We likely won't be able to update this ourselves but if you'd like to submit a fix, we can help review it.

Thanks!

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.