Reduce Unit Test Times (Part 1)

[mrwyattii]

The time and resources required to run all unit tests on each PR has increased substantially in the past year. We are making an effort to reduce the tests done in order to improve the pace at which we can test and merge PRs.

Changes in this PR:

• Move torch19* tests to nightly tests (i.e., not done for each PR)
• Use a cached Apex build for nv-megatron. We store the build on blob storage and add it to our PYTHONPATH. In the case that Apex is updated to a new version, we will upgrade the cached build.

Changes for future PRs:

• nv-inference
  • Utilize coverage to reduce the total number of tests to a bare minimum that touches all inference code
  • Reduce number of parameterize options
• nv-mii
  • Add a @pytest.mark.short marker to the MII unit tests
  • Test only a subset of all MII unit tests
• nv-torch-latest-v100
  • Use coverage to reduce total number of tests and parameterized values
• Add email notifications to the team when nightly tests fail
• Add a coverage report