Question on T5 NLG

[AtheerAlgherairy]

I have a question regarding the training loss for T5 NLG. If we do not set 'metric_for_best_model' to 'bleu,' as shown in the picture below, is it automatically set to 'loss'? What is the best practice for training T5 NLG?

[zqwerty]

Yes, the default metric is loss. According to some NLG studies/practices, continually training the model after achieving the lowest validation loss can still improve metrics like BLEU.

[AtheerAlgherairy]

Thanks..

[AtheerAlgherairy]

Hi.. I used T5-base for NLG.. I got the following results..

However, I got 'err': 0.5966753105391215 . Any idea how to improve the "err"?

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10.0

Framework versions

• Transformers 4.24.0
• Pytorch 2.0.1+cu118
• Datasets 2.7.1
• Tokenizers 0.13.2

[zqwerty]

Sorry for the late reply. Does "err" mean slot error rate? Maybe you could try some pre-training?