Submit ray job after cluster is ready #405
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
|
Actually this one looks straightforward, I can merge it and update #398 |
|
The master RayJob Controller needs lots of improvement. I think after this one and #398, it should be good |
Jeffwan
approved these changes
Jul 25, 2022
| if rayClusterInstance.Status.State != rayv1alpha1.Ready { | ||
| r.Log.Info("waiting for the cluster to be ready", "rayCluster", rayClusterInstance.Name) | ||
| err = r.updateState(ctx, rayJobInstance, rayv1alpha1.JobDeploymentStatusInitializing, nil) | ||
| return ctrl.Result{}, err |
There was a problem hiding this comment.
minor: We can requeue it after x seconds since cluster ready takes a while. without requeueAfter we will see lots of duplicated logs.
This is minor, I can update it later in another PR
lowang-bh
pushed a commit
to lowang-bh/kuberay
that referenced
this pull request
Sep 24, 2023
* Submit ray job after cluster is ready Signed-off-by: Kevin Su <[email protected]> * Fix test errors Signed-off-by: Kevin Su <[email protected]> * Fix test errors Signed-off-by: Kevin Su <[email protected]> * Fix test errors Signed-off-by: Kevin Su <[email protected]> * Fix test errors Signed-off-by: Kevin Su <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Kevin Su [email protected]
Why are these changes needed?
Ray-operator try to submit the ray job when the ray cluster is not ready. we can check the cluster state first before submitting the job.
Before
2022-07-23T08:44:48.609Z INFO raycluster-controller reconcilePods {"all workers already exist for group": "test-group"} 2022-07-23T08:44:48.755Z ERROR controller.rayjob Reconciler error {"reconciler group": "ray.io", "reconciler kind": "RayJob", "name": "test2-n0-0", "namespace": "flytesnacks-development", "error": "Get \"http://test2-n0-0-raycluster-fm75r-head-svc.flytesnacks-development.svc.cluster.local:8265/api/jobs/test2-n0-0-f4nh7\": dial tcp 10.96.63.5:8265: connect: connection refused"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 2022-07-23T08:44:48.756Z INFO raycluster-controller reconciling RayJob {"NamespacedName": "flytesnacks-development/test2-n0-0"} 2022-07-23T08:44:48.756Z INFO controllers.RayJob getOrCreateRayClusterInstance {"rayClusterInstanceName": "test2-n0-0-raycluster-fm75r"} 2022-07-23T08:44:48.855Z ERROR controller.rayjob Reconciler error {"reconciler group": "ray.io", "reconciler kind": "RayJob", "name": "test2-n0-0", "namespace": "flytesnacks-development", "error": "Get \"http://test2-n0-0-raycluster-fm75r-head-svc.flytesnacks-development.svc.cluster.local:8265/api/jobs/test2-n0-0-f4nh7\": dial tcp 10.96.63.5:8265: connect: connection refused"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227After
2022-07-24T02:21:57.951Z INFO raycluster-controller reconciling RayJob {"NamespacedName": "flytesnacks-development/achss4th7xtc5mh6hm27-n0-0"} 2022-07-24T02:21:57.951Z INFO controllers.RayJob getOrCreateRayClusterInstance {"rayClusterInstanceName": "achss4th7xtc5mh6hm27-n0-0-raycluster-sqfvh"} 2022-07-24T02:21:57.952Z INFO controllers.RayJob waiting for the cluster to be ready {"rayCluster": "achss4th7xtc5mh6hm27-n0-0-raycluster-sqfvh"}Related issue number
Checks