PyTorchJob Pods "Not Ready" After Completing Training

[TrevorM15]

I have tried a couple different models (yolox and convmixer) with the PyTorchJob CRD. I watch the logs from the master and worker pods and see the training progress and complete, but when training is done and the logs finish the pods go from the "Running" status to "Not Ready." I would expect them to go to "Completed" status so my pipeline that I've made to train and serve the model can progress, but instead it times-out and fails. I am running Kubeflow v1.5 on EKS and the latest version of training-operator.

[TrevorM15]

I was putting the istio sidecar annotation in the wrong spot.