cluster status safeguards

[Maxusmusti]

Issue link

#250

What changes have been made

Added error handling to prevent cluster.status/wait_ready crashes due to missing AW status

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[kpouget]

@Maxusmusti , is there an easy way to test this PR? ideally, something I could put there to confirm that it does the trick?

[MichaelClifford]

Tested it in my ODH environment and appears to overcome the issue with wait_ready() failing

[Maxusmusti]

@kpouget This should hopefully fix the MissingModel issue you were running into. So immediately running cluster.up() -> cluster.status() or cluster.up() -> cluster.wait_ready() should never crash or error out on the second line. To give MCAD a bit more load, you can run:

cluster.up()
cluster.down()
cluster.up()
cluster.wait_ready()

and it should never crash throughout the wait_ready()

You can install this specific fork/branch in a couple of ways, either clone the repo (checkout correct branch) and run pip install -e . or you can pip install directly from git: https://stackoverflow.com/questions/20101834/pip-install-from-git-repo-branch

[kpouget]

@Maxusmusti, I cannot test this PR because #255 is blocking my automation.

If you want to try it while I'm away, make this comment in openshift-psap/ci-artifacts#876

test codeflare-light

then navigate to this path:
/artifacts/e2e/test/artifacts/000__sdk_user_run_many/000__local_ci__run_multi/ci-pods_artifacts/ci-pod-0/run.log
to have the logs of pod execution

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MichaelClifford

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [MichaelClifford]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[MichaelClifford]

/LGTM