[apiserver] RayJob creation gives an error "Namespace is empty", even after clearly specifying the namespace in params

[shubhscoder]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

apiserver

What happened + What you expected to happen

I am following this tutorial to create a RayJob using the apiserver: https://ray-project.github.io/kuberay/components/apiserver/#create-ray-job-in-a-given-namespace

First, I got this issue:

“{"code":12,"message":"Method Not Allowed"}”

I realized that the url specified in the docs is incorrect and changing it worked for me.

However, now I am running into another issue:

{"code":3,"message":"Namespace is empty. Please specify a valid value.: Validate job request failed.: Invalid input error: Namespace is empty. Please specify a valid value."}

The expectation is the job gets created / fails some other error.

Reproduction script

curl -X POST 'localhost:31888/apis/v1alpha2/jobs' --header 'Content-Type: application/json' --data @ray_cron_job_params.json

Required params in ray_cron_job_params.json

"Name": "rayjob-sample", "Namespace": "ray-system", "user": "test", .....

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

cc @tedhtchang @blublinsky @anishasthana @z103cb

[kevin85421]

Hi @shubhscoder, thank you for reporting this issue! The KubeRay API Server hasn’t been maintained for several quarters, so I wouldn’t recommend using it at the moment. However, recently, folks from IBM/RedHat have been planning to improve it and integrate the component into their production environments. You can expect significant improvements within a month.

[tedhtchang]

Based on the swagger definition
Could you try namespace in the query. For example curl -X POST 'localhost:31888/apis/v1alpha2/jobs? namespace=xxxxx

This issue is similar. We can update the document first if that helps

[tedhtchang]

This worked for me:

curl -X POST 'localhost:31888/apis/v1alpha2/namespaces/default/compute_templates' \
--data '{
  "name": "default-template",
  "namespace": "default",
  "cpu": 1,
  "memory": 1
}'

curl -X POST 'http://localhost:31888/apis/v1alpha2/jobs?namespace=default' \
--data '{
  "name": "rayjob-test",
  "namespace": "default",
  "user": "tedchang",
  "entrypoint": "python -V",
  "clusterSpec": {
    "headGroupSpec": {
      "computeTemplate": "default-template",
      "image": "rayproject/ray:2.5.0",
      "serviceType": "NodePort",
      "rayStartParams": {"dashboard-host": "0.0.0.0"}
    },
    "workerGroupSpec": [
      {
        "groupName": "small-wg",
        "computeTemplate": "default-template",
        "image": "rayproject/ray:2.5.0",
        "replicas": 1,
        "minReplicas": 0,
        "maxReplicas": 1,
        "rayStartParams": {"metrics-export-port": "8080"}
      }
    ]
  }
}'

[blublinsky]

Yea, this is just a documentation bug. I answered this question a week ago

[shubhscoder]

Hi @shubhscoder, thank you for reporting this issue! The KubeRay API Server hasn’t been maintained for several quarters, so I wouldn’t recommend using it at the moment. However, recently, folks from IBM/RedHat have been planning to improve it and integrate the component into their production environments. You can expect significant improvements within a month.

Hello @kevin85421 Thanks for your response! I was trying to add an example for #1206. But I think it would be better to add the example when things are stabilised on the APIserver side? Or should I go ahead and add an example?

[anishasthana]

I think it's worth adding the example right now -- we can update it later if necessary.

[tedhtchang]

Yea, this is just a documentation bug. I answered this question a week ago

@blublinsky
I think we should update the misleading apiserver documentation first since two people reported the same issue.

[z103cb]

@tedhtchang I have some free cycles, I will take a closer look at this issue. I have updated the swagger ui in #1387. @kevin85421 Can you please assign this ticket to me ?

[tedhtchang]

@z103cb We should also re-gen job.swagger.json to make required true. I have no idea how to do that.

[tedhtchang]

Another related comment. The cluster.swagger.json says RayStartParams are optional but ran into error without them:

{
  "code": 13,
  "message": "Internal Server Error: Create Job failed.: InternalServerError: Failed to create a job for (default/): RayJob.ray.io \"rayjob-test\" is invalid: spec.rayClusterSpec.workerGroupSpecs.rayStartParams: Required value"
}

[z103cb]

@tedhtchang adding [(google.api.field_behavior) = REQUIRED] to the field in the .proto, should do the trick according to this page.

I am testing changes to the swagger / proto using that technique. Will submit a PR to address marking the "required" fields in swagger using the stuff above.

I have fixed the documentation issue for this and other issues in #1435. @shubhscoder please feel free to review that PR.