Add io feature to OTF

[YuuuXie]

Summary

• Feature: add write_model=4 in otf. This mode will back up all checkpoint files after each DFT call. For example, the DFT calls are made at step 1, 11, 23, ..., then there will be folders ckpt_1, ckpt_11, ckpt_23 created that in each folder the checkpoint files at that step are stored.

In this way, the OTF can restart from any of the previous DFT steps easily with the files and OTF.from_checkpoint. This is helpful when the OTF training sometimes gets unphysical after some point in the simulation, e.g. a certain DFT fails to converge but the MD still goes on with the unconverged data, or the gp hyps optimization fails to converge, etc.

• Other updates in otf output:
  • recording package version numbers in the header of output file
  • recording per-species MAEs after each DFT call
  • recording DFT data into an .xyz file including sparse environments (target_atoms indices). This .xyz file will include all and only the training frames (and their corresponding sparse environment indices). After the training is done (or when we want to build a master potential from multiple OTFs), it might be easier to build SGP from this file instead of parsing the otf trajectory. Also if we want to release a training data set or give training data to other people (e.g. the NN folks), we can use this file or combine multiple .xyz files from different OTFs.
  • updating the OTF parser

Checklist

Before a pull request can be merged, the following items must be checked:

• Code is in the standard Python style.
  Run Black on your local machine.
• Docstrings have been added in the Sphinx docstring format.
• Type annotations are highly encouraged.
• Tests have been added for any new functionality or bug fixes.
• All existing tests pass.
• The version number in setup.py is updated. We are using a version number format a.b.c
  • If this PR fixes bugs, update version number to a.b.c+1
  • If this PR adds new features, update version number to a.b+1.0
  • If this PR includes significant changes in framework or interface, update version number to a+1.0.0

Note that the CI system will run all the above checks. But it will be much more
efficient if you already fix most errors prior to submitting the PR.

[jonpvandermause]

Looks good to me, main comment is to make sure the PR description gives a complete summary of the changes (e.g. recording per-species MAEs and version numbers in the output file, recording DFT data and sparse environments, updating the OTF parser, etc.)

[YuuuXie]

Looks good to me, main comment is to make sure the PR description gives a complete summary of the changes (e.g. recording per-species MAEs and version numbers in the output file, recording DFT data and sparse environments, updating the OTF parser, etc.)

Oh yes absolutely, thanks for reviewing and pointing this out! The descriptions are updated

[jonpvandermause]

Looks good to me, main comment is to make sure the PR description gives a complete summary of the changes (e.g. recording per-species MAEs and version numbers in the output file, recording DFT data and sparse environments, updating the OTF parser, etc.)

Oh yes absolutely, thanks for reviewing and pointing this out! The descriptions are updated

Looks good! Feel free to merge