[Wait for #2562 ] [ Weight ] Add Var32 Tensor in Weight @open sesame 05/03 15:14 #2563
Conversation
jijoongmoon
requested review from
myungjoo,
again4you,
jaeyun-jung,
leemgs,
wooksong,
helloahn,
kparichay,
gichan-jang,
anyj0527,
zhoonit,
lhs8928,
songgot,
jihochu,
DonghakPark and
SeoHyungjun
as code owners
|
📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2563. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/. |
jijoongmoon
requested review from
baek2sm,
skykongkong8,
djeong20,
EunjuYang and
a team
as code owners
jijoongmoon
changed the title
|
|
cibot:
jijoongmoon
changed the title
It add loss scale property as model common property. Signed-off-by: Jiho Chu <[email protected]>
This PR enables the loss scale factor in Weight. . Change the WeightSpec to incluide the loss factor . Add LossScaleForMixed Property as an layer common property, so that it can set the scale factor in initContext. . Add Loss Scale in initContext . Set the LossScaleForMixed Property when there is LossScale Model Property Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
This PR split the Variable and Gradient Dim in Var_Grad and Weight. By this way we can set different Variable Type and Gradient in Wegiht. . add dim_g for gradient in WeightSpec. . manager need to update to support WeightSpec. . Create Tensors according to dim_v and dim_g . Create Weight chaged in Weight.h Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
Loss Scale is more like Rigid Property of model, rather than flexible property. Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
|
|
|
|
|
|
|
|
1 similar comment
|
|
There was a problem hiding this comment.
@jijoongmoon, 💯 All CI checkers are successfully verified. Thanks.
jijoongmoon
changed the title
|
|
There was a problem hiding this comment.
@jijoongmoon, 💯 All CI checkers are successfully verified. Thanks.
We will add Var32 Tensor if the Variable Weight is not Full precision (FP32). This eables the Weight Update with full precision and only Apply Gradient Process ueses this Tensor. Therefore, the lifespan of this tensor should be "ApplyGradient". . Modify TensorPool to generate Weigth considering Mixed Precsion. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
There was a problem hiding this comment.
@jijoongmoon, 💯 All CI checkers are successfully verified. Thanks.
| TensorDim var32_dim(dim_v); | ||
| var32_dim.setDataType(ml::train::TensorDim::DataType::FP32); |
There was a problem hiding this comment.
would it be convenient for you to have a new TensorDim constructor that takes data type?
(e.g., TensorDim var32_dim(dim_v, ml::train::TensorDim::DataType::FP32))
If so, I'll create a new PR with the change.
| TensorDim var32_dim(dim_v); | ||
| var32_dim.setDataType(ml::train::TensorDim::DataType::FP32); | ||
| std::string var32_suffix = ":fp32"; | ||
| std::string var32_name = name + var32_suffix; |
There was a problem hiding this comment.
duplicated with lines 78-79?
| * @return false otherwise | ||
| */ | ||
| bool isMixedPrecision() const { | ||
| return var->getDataType() == ml::train::TensorDim::DataType::FP32; |
There was a problem hiding this comment.
I'm quite confused here. if this function returns true when the variable is not in full precision, shouldn't it be != instead of ==?
|
@jijoongmoon Let's rebase and move on. |
|
closed by #2663 |
In this PR
We will add Var32 Tensor if the Variable Weight is not Full
precision (FP32). This eables the Weight Update with full precision
and only Apply Gradient Process ueses this Tensor. Therefore, the
lifespan of this tensor should be "ApplyGradient".
Also, modify TensorPool to generate Weigth considering Mixed Precision.
Self evaluation:
Signed-off-by: jijoong.moon [email protected]