[Wait for #2567] [ Test ] Mixed Precision Test Case #2568
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We will add Var32 Tensor if the Variable Weight is not Full precision (FP32). This eables the Weight Update with full precision and only Apply Gradient Process ueses this Tensor. Therefore, the lifespan of this tensor should be "ApplyGradient". . Modify TensorPool to generate Weigth considering Mixed Precsion. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
This pr create the variable fp32 tensor when we create the Weight and Optimizer Weight. . update the manager to create Weight with var32 tensor which requested to weight pool. . update the weight requests with Weight Spec and var, grad and var32 tensors which created already. . add clone Tensor with specific type in tensor.h Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
This PR enables the FP16 support for the layers below: . input layer . mse loss layer Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
jijoongmoon
requested review from
myungjoo,
again4you,
jaeyun-jung,
leemgs,
wooksong,
helloahn,
kparichay,
gichan-jang,
anyj0527,
zhoonit,
lhs8928,
songgot,
jihochu,
DonghakPark,
SeoHyungjun,
baek2sm,
skykongkong8,
djeong20,
EunjuYang and
a team
as code owners
|
📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2568. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/. |
|
|
cibot: 
|
|
jijoongmoon
force-pushed
the
mixed_test
branch
2 times, most recently
from
49f4996
to
b51af11
Compare
DonghakPark
reviewed
May 7, 2024
| is_inplace = true; | ||
|
|
||
| /** | ||
| * @note Input Layer assuems that the FP32 IN Tensor always. Therefore, if the |
taos-ci
approved these changes
May 7, 2024
There was a problem hiding this comment.
@jijoongmoon, 💯 All CI checkers are successfully verified. Thanks.
DonghakPark
approved these changes
May 8, 2024
This PR includes the mixed precision test case. . Input - FC - MSE : "batch_size=2", "model_tensor_type=FP16-FP16", "loss_scale=128" **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
taos-ci
approved these changes
May 8, 2024
There was a problem hiding this comment.
@jijoongmoon, 💯 All CI checkers are successfully verified. Thanks.
This commit modify apply gradient in optimizer. We do not need to save optimizer variables in weight type. Only Optimizer needs the optimizer variables and we should update the weight with full precision to maintain the accuracy. Therefore, remove the var32 tensors for optimizer variables. Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
taos-ci
approved these changes
May 9, 2024
There was a problem hiding this comment.
@jijoongmoon, 💯 All CI checkers are successfully verified. Thanks.
DonghakPark
added a commit
to DonghakPark/nntrainer
that referenced
this pull request
May 10, 2024
This PR is to update the mixed precision layer. - integrate nnstreamer#2568 & nnstreamer#2455 - will update more test **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
DonghakPark
added a commit
to DonghakPark/nntrainer
that referenced
this pull request
May 10, 2024
This PR is to update the mixed precision layer. - integrate nnstreamer#2568 & nnstreamer#2455 - will update more test **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
DonghakPark
added a commit
to DonghakPark/nntrainer
that referenced
this pull request
May 10, 2024
This PR is to update the mixed precision layer. - integrate nnstreamer#2568 & nnstreamer#2455 - will update more test **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
DonghakPark
added a commit
to DonghakPark/nntrainer
that referenced
this pull request
May 10, 2024
This PR is to update the mixed precision layer. - integrate nnstreamer#2568 & nnstreamer#2455 - will update more test **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
DonghakPark
added a commit
to DonghakPark/nntrainer
that referenced
this pull request
May 10, 2024
This PR is to update the mixed precision layer. - integrate nnstreamer#2568 & nnstreamer#2455 - will update more test **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
DonghakPark
added a commit
to DonghakPark/nntrainer
that referenced
this pull request
May 16, 2024
This PR is to update the mixed precision layer. - integrate nnstreamer#2568 & nnstreamer#2455 - will update more test **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
DonghakPark
added a commit
to DonghakPark/nntrainer
that referenced
this pull request
May 17, 2024
This PR is to update the mixed precision layer. - integrate nnstreamer#2568 & nnstreamer#2455 - will update more test **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
DonghakPark
added a commit
to DonghakPark/nntrainer
that referenced
this pull request
May 27, 2024
This PR is to update the mixed precision layer. - integrate nnstreamer#2568 & nnstreamer#2455 - will update more test **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghak PARK <[email protected]>
EunjuYang
reviewed
Jul 10, 2024
| @@ -1680,6 +1680,13 @@ class Tensor { | |||
| */ | |||
| Tensor clone() const; | |||
|
|
|||
| /** | |||
| * @brief Convient wrapper for inplace copy of @a this. | |||
There was a problem hiding this comment.
Suggested change
| * @brief Convient wrapper for inplace copy of @a this. | |
| * @brief Convenient wrapper for inplace copy of @a this. |
Is it typo? Do you mean convenient ?
EunjuYang
reviewed
Jul 10, 2024
| @@ -114,6 +114,7 @@ class Weight : public Var_Grad { | |||
| * | |||
| * @param v Already created variable object | |||
| * @param g Already created gradient object | |||
| * @param v32 Already created gradient object | |||
There was a problem hiding this comment.
Suggested change
| * @param v32 Already created gradient object | |
| * @param v32 Already created variable32 object |
|
closed by #2663 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this PR
This PR includes the mixed precision test case.
. Input - FC - MSE
: "batch_size=2", "model_tensor_type=FP16-FP16", "loss_scale=128"
Self evaluation:
Signed-off-by: jijoong.moon [email protected]