Refactor ray creation

[Bobbins228]

Issue link

Closes: RHOAIENG-10385 and RHOAIENG-8846

What changes have been made

• Removed base_template.yaml
• Removed generate_yaml.py in favour of build_ray_cluster.py
• Removed unneeded labels
• Removed head_info
• Removed machine_types
• Changed app_wrapper_yaml to resource_yaml and removed app_wrapper_name
• Made ray_version a variable for potential future automation
• Moved is_openshift_cluster function to cluster.py
• Changed create_app_wrapper to create_resource
• Removed enableIngress catch logic
• Removed app_wrapper_name
• Removed _components_resources_down in favour of _delete_resources
• Refactored the get_cluster method to generate a new ClusterConfiguration with just the name and namespace of the cluster and retrieved yaml.
• The env config param now actually works

Verification steps

Setup

Notebook server ODH/RHOAI/Local

• Clone this repository with git clone https://github.com/project-codeflare/codeflare-sdk.git
• Checkout this PR's branch
• Run poetry build - install if needed (pip install poetry)
• Run pip install --force-reinstall dist/codeflare_sdk-0.0.0.dev0-py3-none-any.whl
• Restart your notebook kernel

Testing

All ClusterConfiguration parameters must be tested with the new cluster creation method.

Keep a special eye out for the following as they were the most complex to implement:

• worker_extended_resource_requests
• head_extended_resource_requests
• extended_resource_mapping
• The notebook annotations.

Recommendation

Have 2 separate virtual envs 1 with main SDK and 1 with this PR's SDK and compare created Ray Clusters.
Small things like blank image pull secrets and some example metadata labels are removed from every generated ray cluster so they wont be an exact match. The important thing to look out for is if the configurations match 👍

Automated Notebook testing should cover the functionality changed but I still suggest all parameters should be human verified.

Test the new and improved get_cluster() function.

NOTE: You can compare the original & retrieved clusters by setting write_to_file=True on ClusterConfiguration and get_cluster()

NOTE 2: get_cluster() will also retrieve the mtls/oauth containers as well. This has no impact on the ability to create the cluster after deleting it through get_cluster() -> cluster.down() -> cluster.up()

• Create a Ray Cluster
• Restart your notebook kernel
• Run cluster = get_cluster(cluster_name=<name>, namespace=<namespace>, write_to_file=True)
• Run through the basic cluster. methods
• Run cluster.down() then cluster.up()
• Ensure the cluster is created successfully
• Repeat steps for AppWrapper (Remember to enable AppWrappers in the CFO configmap!)

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[codecov]

Codecov Report

Attention: Patch coverage is 95.39474% with 14 lines in your changes missing coverage. Please review.

Project coverage is 92.91%. Comparing base (63ee4ae) to head (5f5c2ab).

Files with missing lines	Patch %	Lines
src/codeflare_sdk/ray/cluster/cluster.py	80.30%	13 Missing ⚠️
src/codeflare_sdk/ray/cluster/build_ray_cluster.py	99.40%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #751      +/-   ##
==========================================
- Coverage   94.12%   92.91%   -1.21%     
==========================================
  Files          36       36              
  Lines        2417     2400      -17     
==========================================
- Hits         2275     2230      -45     
- Misses        142      170      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ChristianZaccaria]

This is a great PR, awesome work Mark! :)

I left some minor nitpicks. I'll now give this a run in a cluster.

[ChristianZaccaria]

I noticed when comparing the YAMLs between this PR and main branch, on this PR, the RayCluster yaml generated contains explicitly:
imagePullPolicy: Always

In main, this is unset. By default, if the image tag is :latest the imagePullPolicy is set to Always, otherwise, the default is ifNotPresent. ifNotPresent may be preferred here to only pull the image if it's not cached or doesn't already exist on the node.

[ChristianZaccaria]

• YAMLs look as expected when comparing with main and this PR.
  I tested the following scenarios:
  • LocalQueue set, no image (defaults to py3.9 image on OpenShift).
  • LocalQueue not set, no image (defaults to py3.9 image on OpenShift).
  • Python3.11 environment: the image changes to Ray image for py3.11.
  • Testing most parameters: Set envs, AppWrapper true, no LQ, set custom image, and set gpus.
  • Testing most parameters: Set envs, AppWrapper true, set LQ, set custom image, and set gpus.
• AppWrappers and RayClusters work as expected.
• get_cluster() works well!

/lgtm thanks! Great work!

[Bobbins228]

/retest

[KPostOffice]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChristianZaccaria, KPostOffice

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ChristianZaccaria,KPostOffice]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment