Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ONNX BERT Example and Fixes #4757

Merged
merged 7 commits into from
Aug 11, 2020
Merged

Add ONNX BERT Example and Fixes #4757

merged 7 commits into from
Aug 11, 2020

Conversation

rayankrish
Copy link
Contributor

  • Fix training issue when passing .onnx model to ORTTrainer.
  • Add example using ONNX BERT toy model.
  • Minor refactor by moving debug.py in core to _test_helpers.py in test folder.

Thiago Crepaldi and others added 6 commits August 11, 2020 18:57
Initial implementation for mixed precision/loss scaler. Untested
bb694e9
Add Mixed precision/LossScaler + several fixes
36ed5aa 
Fix CUDA training
Add optimization_step into TrainStepInfo class
Refactor LRSCheduler to use optimization_step instead of step
Updated several default values at ORTTrainerOptions
Add initial Gradient Accumulation supported. Untested
Fix ONNX model post processing
Refactor unit tests
Add unit test for mixed precision and minor fixes
56b55db
bert example amp dtype crash
92a4eb2
bert toy example and fixes
3966eeb
minor refactor
bdee9c3
@rayankrish rayankrish requested a review from a team as a code owner August 11, 2020 19:22
@thiagocrepaldi thiagocrepaldi linked an issue Aug 11, 2020 that may be closed by this pull request
[WIP] New PyTorch frontend API #4176
Closed
@thiagocrepaldi thiagocrepaldi merged commit adb1b67 into microsoft:feature/new_pytorch_frontend Aug 11, 2020
thiagocrepaldi pushed a commit that referenced this pull request Aug 12, 2020
@rayankrish
Add ONNX BERT example + minor fixes (#4757)
fb4c90e 
* Fix training issue when passing ONNX file into ORTTrainer

Co-authored-by: Thiago Crepaldi <[email protected]>
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
thiagocrepaldi pushed a commit that referenced this pull request Aug 14, 2020
@rayankrish
Add ONNX BERT example + minor fixes (#4757)
4fec029 
* Fix training issue when passing ONNX file into ORTTrainer

Co-authored-by: Thiago Crepaldi <[email protected]>
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
@thiagocrepaldi thiagocrepaldi mentioned this pull request Aug 14, 2020
Closed
@thiagocrepaldi thiagocrepaldi removed a link to an issue Aug 14, 2020
[WIP] New PyTorch frontend API #4176
Closed
thiagocrepaldi pushed a commit that referenced this pull request Aug 15, 2020
@rayankrish
Add ONNX BERT example + minor fixes (#4757)
ee1fc01 
* Fix training issue when passing ONNX file into ORTTrainer

Co-authored-by: Thiago Crepaldi <[email protected]>
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
thiagocrepaldi pushed a commit that referenced this pull request Aug 15, 2020
@rayankrish
Add ONNX BERT example + minor fixes (#4757)
4568982 
* Fix training issue when passing ONNX file into ORTTrainer

Co-authored-by: Thiago Crepaldi <[email protected]>
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
thiagocrepaldi pushed a commit that referenced this pull request Aug 17, 2020
@rayankrish
Add new PytTrch front-end (#4815)
42408aa 
* Add ORTTrainerOptions class for the new pytorch frontend (#4382)

Add ORTTrainerOptions class and some placeholders

* Add _ORTTrainerModelDesc to perform validation for model description (#4416)

* Add Loss Scaler classes to the new frontend (#4306)

* Add TrainStepInfo used on the new frontend API (#4256)

* Add Optimizer classes to the new frontend (#4280)

* Add LRScheduler implementation (#4357)

* Add basic ORTTrainer API (#4435)

This PR presents the public API for ORTTrainer for the short term
development.

It also validates and saves input parameters, which will be used in the
next stages, such as building ONNX model, post processing the model and
configuring the training session

* Add opset_version into ORTTrainerOptions and change type of ORTTrainer.loss_fn (#4592)

* Update ModelDescription and minor fix on ORTTrainer ctor (#4605)

* Update ModelDescription and minor fix on ORTTrainer/ORTTrainerOptions

This PR keeps the public API intact, but changes how model description is stored on the backend

Currently, users creates a dict with two lists of tuples.
One list called 'inputs' and each tuple has the following format tuple(name, shape).
The second list is called 'outputs' and each tuple can be either tuple(name, shape) or tuple(name, shape, is_loss).

With this PR, when this dict is passed in to ORTTrainer, it is fully validated as usual.
However, tuples are internally replaced by namedtuples and all output tuples will have
tuple(name, shape, is_loss) format instead of is_loss being optionally present.

Additionally to that normalization in the internal representation (which eases coding),
two internal methods were created to replace a namedtuple(name, shape) to namedtuple(name, shape, dtype)
or namedtuple(name, shape, is_loss, dtype) dependeing whether the tuple is an input or output.

This is necessary as ORTTRainer finds out data types of each input/output during model export to onnx.

Finally, a minor fix was done on ORTTrainer. It could initialize ORTTrainerOptions incorrectly when options=None

* Rename input name for test

* Add ONNX Model Export to New Frontend (#4612)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>

* Create training session + minor improvements (#4668)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>

* Save ONNX model in file (#4671)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>

* Add eval step (#4674)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>

* Add train_step (#4677)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>

* Add LR Scheduler (#4694)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>

* Add deterministic compute tests (#4716)


Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>

* Add legacy vs experimental ORTTrainer accuracy comparison (#4727)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>

* Add Mixed precision/LossScaler + several fixes (#4739)

Additionally to the mixed precision/loss scaler code, this PR includes:

* Fix CUDA training
* Add optimization_step into TrainStepInfo class
* Refactor LRSCheduler to use optimization_step instead of step
* Updated several default values at ORTTrainerOptions
* Add initial Gradient Accumulation supported. Untested
* Fix ONNX model post processing
* Refactor unit tests

* Add ONNX BERT example + minor fixes (#4757)

* Fix training issue when passing ONNX file into ORTTrainer

Co-authored-by: Thiago Crepaldi <[email protected]>
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>

* Add Dynamic Shape support (#4758)

* Update DeepSpeed Zero Stage option to a separate option group (#4772)

* Add support to fetches (#4777)

* Add Gradient Accumulation Steps support (#4793)

* Fix Dynamic Axes feature and add unit test (#4795)

* Add frozen weights test (#4807)

* Move new pytorch front-end to 'experimental' namespace (#4814)

* Fix build

Co-authored-by: Rayan-Krishnan <[email protected]>
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thiagocrepaldi thiagocrepaldi thiagocrepaldi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rayankrish @thiagocrepaldi