Add model MegatronBert #1678
Merged
Add model MegatronBert #1678
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Steffy-zxf
reviewed
Feb 21, 2022
| max_position_embeddings=512, | ||
| hidden_dropout_prob=0.1, | ||
| position_embedding_type="absolute"): | ||
| super().__init__() |
There was a problem hiding this comment.
为了保持与paddlenlp代码风格统一，建议写成：
super(MegatronBertEmbeddings, self).__init__()| help="Path to pre-trained model or shortcut name of model.") | ||
| parser.add_argument( | ||
| "--output_dir", | ||
| default="/root/paddlejob/workspace/output", |
ZHUI
reviewed
Feb 24, 2022
| @@ -0,0 +1,135 @@ | |||
| import argparse | |||
| import paddle | ||
|
|
||
| from paddle.io import DataLoader | ||
| from args import parse_args |
There was a problem hiding this comment.
按照 pep8 规则 import. https://www.python.org/dev/peps/pep-0008/#imports
Imports should be grouped in the following order:
Standard library imports.
Related third party imports.
Local application/library specific imports.
You should put a blank line between each group of imports.
| @@ -0,0 +1,2 @@ | |||
| from .modeling import * | |||
| import paddle | ||
| from paddle import nn | ||
| from .. import PretrainedModel, register_base_model | ||
| import paddle.nn.functional as F |
| return x * F.sigmoid(x) | ||
|
|
||
|
|
||
| def gelu_new(x): |
There was a problem hiding this comment.
https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/nn/functional/gelu_cn.html#gelu
gelu with approximate=True？
There was a problem hiding this comment.
是近似计算，我是参考了fnet和bigbird的写法
ZHUI
previously approved these changes
Mar 10, 2022
| --learning_rate=1e-5 \ | ||
| --output_dir=output/ | ||
| --device=gpu | ||
| --num_train_epochs=2 |
| --learning_rate=1e-5 \ | ||
| --output_dir=output/ | ||
| --device=gpu | ||
| --num_train_epochs=2 |
| ```shell | ||
| python -m paddle.distributed.launch run_glue.py \ | ||
| --task_name=mnli \ | ||
| --output_dir=output/ |
| @@ -0,0 +1,150 @@ | |||
| # Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved. | |||
| "--seed", type=int, default=42, help="random seed for initialization") | ||
| parser.add_argument( | ||
| '--device', | ||
| choices=['cpu', 'gpu'], |
| "--device", | ||
| default="gpu", | ||
| type=str, | ||
| choices=["cpu", "gpu", "xpu", "npu"], |
| @@ -0,0 +1,353 @@ | |||
| # Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved. | |||
| return input_tensor + hidden_states | ||
|
|
||
|
|
||
| # Based on transformers.models.bert.modeling_bert.BertLayer. Added LayerNorm. |
|
|
||
| Args: | ||
| vocab_size (int): | ||
| Vocabulary size of `inputs_ids` in `ConvBertModel`. Also is the vocab size of token embedding matrix. |
| @@ -0,0 +1,102 @@ | |||
| # Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved. | |||
yingyibiao
reviewed
Mar 11, 2022
| from paddle import nn | ||
| import paddle.nn.functional as F | ||
|
|
||
| from ...ops import einsum |
yingyibiao
approved these changes
Mar 13, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add new model MegatronBert
The model weight:
链接：https://pan.baidu.com/s/1DNoxmqxtRiMycHfVnvwJwg
提取码：olie