Skip to content

Commit

Permalink
Update Kubernetes manifest files for deploying ChatQnA
Browse files Browse the repository at this point in the history
Update Kubernetes manifest files for deploying ChatQnA without
GMC.

Signed-off-by: Lianhao Lu <[email protected]>
  • Loading branch information
lianhao committed Jul 23, 2024
1 parent 284d855 commit 998fb02
Show file tree
Hide file tree
Showing 25 changed files with 2,282 additions and 901 deletions.
8 changes: 6 additions & 2 deletions ChatQnA/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,9 +105,13 @@ docker compose -f docker_compose.yaml up -d

Refer to the [NVIDIA GPU Guide](./docker/gpu/README.md) for more instructions on building docker images from source.

## Deploy ChatQnA into Kubernetes on Xeon & Gaudi
## Deploy ChatQnA into Kubernetes on Xeon & Gaudi with GMC

Refer to the [Kubernetes Guide](./kubernetes/manifests/README.md) for instructions on deploying ChatQnA into Kubernetes on Xeon & Gaudi.
Refer to the [Kubernetes Guide](./kubernetes/README.md) for instructions on deploying ChatQnA into Kubernetes on Xeon & Gaudi with GMC.

## Deploy ChatQnA into Kubernetes on Xeon & Gaudi without GMC

Refer to the [Kubernetes Guide](./kubernetes/manifests/README.md) for instructions on deploying ChatQnA into Kubernetes on Xeon & Gaudi without GMC.

## Deploy ChatQnA on AI PC

Expand Down
41 changes: 41 additions & 0 deletions ChatQnA/kubernetes/manifests/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<h1 align="center" id="title">Deploy ChatQnA in Kubernetes Cluster</h1>

> [NOTE]
> The following values must be set before you can deploy:
> HUGGINGFACEHUB_API_TOKEN
> You can also customize the "MODEL_ID" if needed.
> You need to make sure you have created the directory `/mnt/opea-models` to save the cached model on the node where the ChatQnA workload is running. Otherwise, you need to modify the `chatqna.yaml` file to change the `model-volume` to a directory that exists on the node.
## Deploy On Xeon

```
cd GenAIExamples/ChatQnA/kubernetes/manifests/xeon
export HUGGINGFACEHUB_API_TOKEN="YourOwnToken"
sed -i "s/insert-your-huggingface-token-here/${HUGGINGFACEHUB_API_TOKEN}/g" chatqna.yaml
kubectl apply -f chatqna.yaml
```

## Deploy On Gaudi

```
cd GenAIExamples/ChatQnA/kubernetes/manifests/gaudi
export HUGGINGFACEHUB_API_TOKEN="YourOwnToken"
sed -i "s/insert-your-huggingface-token-here/${HUGGINGFACEHUB_API_TOKEN}/g" chatqna.yaml
kubectl apply -f chatqna.yaml
```

## Verify Services

To verify the installation, run the command `kubectl get pod` to make sure all pods are running.

Then run the command `kubectl port-forward svc/chatqna 8888:8888` to expose the ChatQnA service for access.

Open another terminal and run the following command to verify the service if working:

```console
curl http://localhost:8888/v1/chatqna \
-H 'Content-Type: application/json' \
-d '{"messages": "What is the revenue of Nike in 2023?"}'
```
45 changes: 0 additions & 45 deletions ChatQnA/kubernetes/manifests/chaqna-xeon-backend-server.yaml

This file was deleted.

74 changes: 0 additions & 74 deletions ChatQnA/kubernetes/manifests/docsum_gaudi_llm.yaml

This file was deleted.

74 changes: 0 additions & 74 deletions ChatQnA/kubernetes/manifests/docsum_llm.yaml

This file was deleted.

45 changes: 0 additions & 45 deletions ChatQnA/kubernetes/manifests/embedding.yaml

This file was deleted.

Loading

0 comments on commit 998fb02

Please sign in to comment.