[Docs][KubeRay] add a guide for deploying vLLM with RayService #47038
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrewsykim
requested review from
architkulkarni,
maxpumperla,
pcmoritz,
kevin85421 and
a team
as code owners
andrewsykim
force-pushed
the
kuberay-vllm-guide
branch
3 times, most recently
from
fdcf9c4
to
10d4985
Compare
|
@angelinalg @kevin85421 this is ready for review now |
andrewsykim
force-pushed
the
kuberay-vllm-guide
branch
from
10d4985
to
bf42e34
Compare
kevin85421
added
the
kuberay
kevin85421
approved these changes
Sep 4, 2024
andrewsykim
force-pushed
the
kuberay-vllm-guide
branch
from
bf42e34
to
07dd54d
Compare
|
@angelinalg can you review please? |
angelinalg
approved these changes
Sep 9, 2024
There was a problem hiding this comment.
Just some style nits. Please consider using Vale to find these issues in the future. Please excuse any inaccuracies I introduced in my suggestions and correct as needed. Happy to answer any questions you have about the suggestions. Thanks for your contribution!
|
|
||
| ## Prerequisites | ||
|
|
||
| This example downloads model weights from Hugging Face. You will need to complete the following |
There was a problem hiding this comment.
Suggested change
| This example downloads model weights from Hugging Face. You will need to complete the following | |
| This example downloads model weights from Hugging Face. You need to complete the following |
| prerequisites to successfully complete this guide: | ||
| * A [Hugging Face account](https://huggingface.co/) | ||
| * A Hugging Face [access token](https://huggingface.co/docs/hub/security-tokens) with read access to gated repos. | ||
| * Access to the Llama 3 8B model. This usually requires signing an agreement on Hugging Face to access this model. Go to the [Llama 3 model page](https://huggingface.co/meta-llama/Meta-Llama-3-8B) for more details. |
There was a problem hiding this comment.
Suggested change
| * Access to the Llama 3 8B model. This usually requires signing an agreement on Hugging Face to access this model. Go to the [Llama 3 model page](https://huggingface.co/meta-llama/Meta-Llama-3-8B) for more details. | |
| * Access to the Llama 3 8B model. Getting access usually requires signing an agreement on Hugging Face to access this model. Go to the [Llama 3 model page](https://huggingface.co/meta-llama/Meta-Llama-3-8B) for more details. |
| --accelerator=type=nvidia-l4,count=2,gpu-driver-version=latest | ||
| ``` | ||
|
|
||
| This example uses L4 GPUs. Each model replica will use 2 L4 GPUs using vLLM's tensor parallelism. |
There was a problem hiding this comment.
Suggested change
| This example uses L4 GPUs. Each model replica will use 2 L4 GPUs using vLLM's tensor parallelism. | |
| This example uses L4 GPUs. Each model replica uses 2 L4 GPUs using vLLM's tensor parallelism. |
| kubectl create secret generic hf-secret --from-literal=hf_api_token=${HF_TOKEN} --dry-run=client -o yaml | kubectl apply -f - | ||
| ``` | ||
|
|
||
| This secret will be referenced as an environment variable in the RayCluster used in the next steps. |
There was a problem hiding this comment.
Suggested change
| This secret will be referenced as an environment variable in the RayCluster used in the next steps. | |
| This guide references this secret as an environment variable in the RayCluster in the next steps. |
| kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/vllm/ray-service.vllm.yaml | ||
| ``` | ||
|
|
||
| The RayService is configured to deploy a Ray Serve application, running vLLM as the serving engine for the Llama 3 8B Instruct model. The code used in this example can be found [here](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/vllm/serve.py). |
There was a problem hiding this comment.
Suggested change
| The RayService is configured to deploy a Ray Serve application, running vLLM as the serving engine for the Llama 3 8B Instruct model. The code used in this example can be found [here](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/vllm/serve.py). | |
| This step configures RayService to deploy a Ray Serve app, running vLLM as the serving engine for the Llama 3 8B Instruct model. You can find the code for this example [on GitHub](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/vllm/serve.py). |
| TENSOR_PARALLELISM: "2" | ||
| ``` | ||
|
|
||
| Wait for the RayService resource to be ready. You can inspect it's status by running the following command: |
There was a problem hiding this comment.
Suggested change
| Wait for the RayService resource to be ready. You can inspect it's status by running the following command: | |
| Wait for the RayService resource to be ready. You can inspect its status by running the following command: |
|
|
||
| ## Send a prompt | ||
|
|
||
| Once you've confirmed the Ray Serve deployment is healthy, you can establish a port-forwarding session for the Serve application: |
There was a problem hiding this comment.
Suggested change
| Once you've confirmed the Ray Serve deployment is healthy, you can establish a port-forwarding session for the Serve application: | |
| Confirm the Ray Serve deployment is healthy, then you can establish a port-forwarding session for the Serve app: |
| $ kubectl port-forward svc/llama-3-8b-serve-svc 8000 | ||
| ``` | ||
|
|
||
| Note that this Kubernetes Service will be created after the Serve applications are ready and running. |
There was a problem hiding this comment.
Suggested change
| Note that this Kubernetes Service will be created after the Serve applications are ready and running. | |
| Note that KubeRay creates this Kubernetes Service after the Serve apps are ready and running. |
Signed-off-by: Andrew Sy Kim <[email protected]>
andrewsykim
force-pushed
the
kuberay-vllm-guide
branch
from
07dd54d
to
f09d4a6
Compare
|
Thanks for the review @angelinalg, addressed your feedback! |
ujjawal-khare
pushed a commit
to ujjawal-khare-27/ray
that referenced
this pull request
Oct 15, 2024
…roject#47038) Signed-off-by: Andrew Sy Kim <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Based on sample in ray-project/kuberay#2289
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.