How can I get mae and rmse of validation and training

[turbosonics]

Hello,

I hope to check mae and rmse of E, F, and stress (and maybe virial too) from validation set and training set. But I don't know how to set such output option. Following is the MACE input setting I'm using for my tests;

mace_run_train \
    --name="Test" \
    --seed=123 \
    --device=cuda \
    --default_dtype="float32" \
    --error_table="PerAtomMAE" \
    --model="MACE" \
    --r_max=6.0 \
    --hidden_irreps='128x0e + 128x1o' \
    --num_channels=32 \
    --max_L=2 \
    --compute_stress=True \
    --compute_forces=True \
    --train_file="${GEOMETRY_DIR}/${GEOMETRY_TRAIN_NAME}" \
    --valid_fraction=0.3 \
    --test_file="${GEOMETRY_DIR}/${GEOMETRY_TEST_NAME}" \
    --E0s="average" \
    --energy_key="energy" \
    --forces_key="forces" \
    --stress_key="stress" \
    --loss="weighted" \
    --forces_weight=1.0 \
    --energy_weight=1.0 \
    --stress_weight=1.0 \
    --config_type_weights='{"Default":1.0}' \
    --optimizer="adam" \
    --batch_size=5 \
    --valid_batch_size=5 \
    --lr=0.005 \
    --amsgrad \
    --scheduler="ReduceLROnPlateau" \
    --lr_factor=0.5 \
    --scheduler_patience=50 \
    --lr_scheduler_gamma=0.99 \
    --ema \
    --ema_decay=0.99 \
    --max_num_epochs=500 \
    --patience=50 \
    --restart_latest \
    --save_cpu \
    --clip_grad=100.0 \

And this is what I'm getting from result:

...
{"loss": 0.0002929910842794925, "time": 0.2666587829589844, "mode": "opt", "epoch": 498}
{"loss": 0.002309008482221874, "mae_e": 0.15782718360424042, "mae_e_per_atom": 0.0007970419828779995, "rmse_e": 0.22078144550323486, "rmse_e_per_atom": 0.0011314523871988058, "q95_e": 0.47121582031249964, "mae_f": 0.024527879431843758, "rel_mae_f": 3.2211644996095816, "rmse_f": 0.04782024770975113, "rel_rmse_f": 3.510639968794681, "q95_f": 0.09724280387163162, "mae_stress": 0.0073347738943994045, "rmse_stress": 0.01363541092723608, "rmse_stress_per_atom": 7.09116502548568e-05, "q95_stress": 0.029036216065287588, "mae_virials": 62.1697883605957, "rmse_virials": 129.260009765625, "rmse_virials_per_atom": 0.6706804037094116, "q95_virials": 393.78586730957005, "time": 113.37074255943298, "mode": "eval", "epoch": 498}
{"loss": 0.00036834419006481767, "time": 0.27051448822021484, "mode": "opt", "epoch": 499}
{"loss": 0.00502742687240243, "time": 0.27277469635009766, "mode": "opt", "epoch": 499}
...

Is this something related to --error_table option? Or --loss option?

Thanks

[pobo95]

--error_table=PerAtomRMSEstressvirials this option maybe be help you

[pobo95]

When you check the log file, MACE seperated validation set from training set automatily (maybe 20% of training set).

When the training is finished (reach to max_ephoch), it shows RMSE or MAE of test set.

But during the training, it shows RMSE/MAE of validation sets.

If you want to see training sets' RMSE you should change code little (MACE/tools/train.py and MACE/modules/loss.py .etc..)

[turbosonics]

Please correct me if I'm wrong, but from what I know about graph neural network training in general, we need to compare errors (RMSE and MAE) between validation set and training set to make a decision on overfitting and underfitting. Right?

I think I'm doing something wrong, but I still don't get it. Why users need to change the code to check RMSE and MAE of loss functions for training set and validation set to judge overfitting and underfitting?... Wouldn't it be better for MACE if the code displays them from the beginning from log and output file?

Could you let me know how did you decide overfitting or underfitting of given training data set from AIMDs, without modifying codes, from MACE training?

[ilyes319]

You can look at the training error in the results folder, it saves a training log for each batch.

In "modern neural networks" the loss value will go down close to zero for most dataset. The best to monitor overfitting is to check if the validation loss increases.

[pobo95]

@turbosonics
Actually you don't need to check training set's RMSE/MAE.

If overfitting occurs, it cannot be distinguished only by the RMSE/MAE of the training set.
This is because the loss decreases regardless of overfitting.

However, overfitting can be confirmed with the value of validation as @ilyes319 said.

In case of underfitting
RMSE/MAE of the training set will increase and the value of validation set will also increase.

Therefore, you only need to check the RMSE/MAE of the validation set.
If it increases, something is going wrong.

[turbosonics]

@ilyes319 @pobo95
Thanks for your suggestions.

I don't have huge amount of GNN model training experience. But from other GNN model training project for different applications (it was not about GNNFF MD), I experienced some cases where loss has declined close to zero and keep nice steady state, yet the model is declared as overfitting because validation set RMSE (and MAE) of a single loss function was clearly higher than training set RMSE. But all other curves were fine. Back then, we concluded the training data had a lot of noise which caused such behavior, so we used several different measures to bypass...

For case of GNNFF MD framework, I still think such scenarios might be possible (but I might be wrong). We could think of an example scenario of fine-tuning (or from the scratch) training where AIMD training data contains a lot of various configurations such as surface reconstruction, interface behavior, NPT ensembles of high temperature, structural deformation of various crystal and/or non-crystals...

But if MACE can digest those, then I will be happy! Also, if there is any training strategy/approach that helps to train such AIMD data using MACE, I really hope to learn more about this!

Thanks!

PS) Oh, and please forgive my ignorance, but could anyone explain me what is "test set" of MACE means? Is this means the "validation set" of typical other GNN model training? I understand that way but please correct me if I was wrong.

[gabor1]

I would certainly hope that MACE can "digest" any reasonable configuration.

[gabor1]

Test set is an entirely independent set that you can specify and RMSEs get printed for it, but it is not used in any way in training. Validation set is used to stop the training.

[nihil39]

Hi,

So the error for validation set is in the Log folder while the error for training and test set (if specified) is in the Results folder, right?

[ilyes319]

The validation and test errors are in the log file and the train errors and additional metrics on the validation are in the results folder. I recommend looking mostly at the log file.