[PyTorch] Reorganize L1 tests

[timmoon10]

Description

Our testing scheme currently looks like:

• L0: run in all CI pipelines and with nightly builds
• L1: monthly builds
• L2: ?
• L3: ?

I propose the following:

• L0: run in all CI pipelines
• L1: nightly builds
• L2: weekly builds
• L3: monthly builds

This PR consolidates the distributed PyTorch tests into L1 and moves the GPT convergence test to L3.

Note that we're not really following the standard testing nomenclature since most of our Python L0 tests should be classified as L1 or L2. This matches our workflow though, so I don't see a need to be pedantic.

Type of change

• Documentation change (change only to the documentation, either a fix or a new content)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Infra/Build change
• Code refractor
• Testing

Changes

• Consolidate PyTorch distributed tests within single L1 test script
• Move PyTorch GPT convergence test to L3 test script

Checklist:

• I have read and followed the contributing guidelines
• The functionality is complete
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[pggPL]

LGTM

Can we add some way to run L1 tests from github?

[timmoon10]

Pipeline 19429427 seems to work as expected. The test failures don't seem related to this PR.

[timmoon10]

Pipeline 19461229