[AutoParallel] Add benchmark for llama-7b-dy2st. #8559
Conversation
|
Thanks for your contribution! |
GhostScreaming
force-pushed
the
llama7b_benchmark
branch
from
e173eb7
to
423cd8e
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #8559 +/- ##
===========================================
- Coverage 53.86% 53.86% -0.01%
===========================================
Files 620 620
Lines 97081 97110 +29
===========================================
+ Hits 52296 52304 +8
- Misses 44785 44806 +21 ☔ View full report in Codecov by Sentry. |
| "per_device_eval_batch_size": 2, | ||
| "tensor_parallel_degree": 1, | ||
| "pipeline_parallel_degree": 1, | ||
| "sharding": "stage1", |
There was a problem hiding this comment.
动态图stage1跟静态图stage2一致。
| "tensor_parallel_degree": 1, | ||
| "pipeline_parallel_degree": 1, | ||
| "sharding": "stage1", | ||
| "sharding_parallel_config": "enable_stage1_overlap", |
There was a problem hiding this comment.
"enable_stage1_overlap" --> "enable_stage2_overlap"
| "pipeline_parallel_degree": 1, | ||
| "sharding": "stage1", | ||
| "sharding_parallel_config": "enable_stage1_overlap", | ||
| "tensor_parallel_config": "enable_delay_scale_loss enable_mp_async_allreduce enable_mp_skip_c_identity enable_mp_fused_linear_param_grad_add", |
There was a problem hiding this comment.
"tensor_parallel_config": "enable_delay_scale_loss enable_mp_async_allreduce enable_mp_skip_c_identity enable_mp_fused_linear_param_grad_add",
-->
"tensor_parallel_config": "enable_mp_async_allreduce",
| "sharding": "stage1", | ||
| "sharding_parallel_config": "enable_stage1_overlap", | ||
| "tensor_parallel_config": "enable_delay_scale_loss enable_mp_async_allreduce enable_mp_skip_c_identity enable_mp_fused_linear_param_grad_add", | ||
| "pipeline_parallel_config": "enable_delay_scale_loss enable_release_grads disable_partial_send_recv", |
There was a problem hiding this comment.
没有开PP并行，"pipeline_parallel_config": ""设置为空即可
| "weight_decay": 0.01, | ||
| "bf16": true, | ||
| "fp16_opt_level": "O2", | ||
| "amp_master_grad": true, |
There was a problem hiding this comment.
增加：
"amp_custom_black_list": ["reduce_sum", "c_softmax_with_cross_entropy"],
"amp_custom_white_list": ["lookup_table", "lookup_table_v2"],
PR types
Others
PR changes
Others
Description
Add llama2-7b for test_tipc