[Bug Fix] fix paddle multipy_fwd_func warning message

[BeingGod]

PR types

Bug fixes

PR changes

Others

Description

由于主框架在 PaddlePaddle/Paddle#59518 中对于multiply输入类型不一致会输出warning日志。
在amp level=O2情况下：
hidden_states = paddle.rsqrt(variance + self.variance_epsilon) * hidden_states中paddle.rsqrt(variance + self.variance_epsilon)结果为float32，hidden_states 为fp16/bf16。

报错日志

复现脚本：

SCRIPT_HOME=$(cd $(dirname $0); pwd)

CARDS="0,1"

task_name="llama_pp2dp4"
rm -rf "$SCRIPT_HOME/output/$task_name/"
rm -rf "$SCRIPT_HOME/output/${task_name}_log"

TP=2
PP=1
SHARDING_STAGE="stage1"


python -u  -m paddle.distributed.launch \
    --devices=$CARDS \
    --log_dir "output/$task_name""_log" \
    run_pretrain.py \
    --model_name_or_path "__internal_testing__/tiny-random-llama" \
    --tokenizer_name_or_path "__internal_testing__/tiny-random-llama" \
    --input_dir "/workspace/dataset/llama_openwebtext2" \
    --output_dir "output/$task_name" \
    --split 949,50,1 \
    --max_seq_length 1024 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --fuse_attention_qkv 1 \
    --fuse_attention_ffn 1 \
    --use_flash_attention 1 \
    --use_fused_rms_norm 0 \
    --fp16 \
    --fp16_opt_level "O2" \
    --scale_loss 1024 \
    --amp_master_grad 1 \
    --max_grad_norm 1.0 \
    --tensor_parallel_degree $TP \
    --pipeline_parallel_degree $PP \
    --sharding $SHARDING_STAGE \
    --learning_rate 5.0e-5 \
    --min_learning_rate 1.0e-9 \
    --lr_scheduler_type "cosine" \
    --max_steps 6000 \
    --save_steps 6000 \
    --weight_decay 0.01 \
    --adam_beta1 0.9 \
    --adam_beta2 0.95 \
    --warmup_ratio 0.1 \
    --logging_steps 1 \
    --dataloader_num_workers 0 \
    --gradient_accumulation_steps 1 \
    --eval_steps 1000 \
    --report_to "visualdl" \
    --disable_tqdm true \
    --continue_training 0 \
    --recompute 0 \
    --do_train \
    --device "gpu" \
    --overwrite_output_dir True

[paddle-bot]

Thanks for your contribution!

[ZHUI]

LGTM

[codecov]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (5c2bf81) 0.00% compared to head (b893463) 57.11%.
Report is 1 commits behind head on develop.

Files	Patch %	Lines
paddlenlp/transformers/llama/modeling.py	50.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #7818       +/-   ##
============================================
+ Coverage         0   57.11%   +57.11%     
============================================
  Files            0      587      +587     
  Lines            0    88196    +88196     
============================================
+ Hits             0    50377    +50377     
- Misses           0    37819    +37819     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.