Bug Fix: chat completions API calls need model_id

[tybrs]

Description

Users must pass model parameter explicitly to use /v1/chat/completions API call:

curl http://localhost:8088/v1/chat/completions \
    -X POST \
    -d '{"model": "meta-llama/Meta-Llama-Guard-2-8B", "messages": [{"role": "user", "content": "Say this is a test!"}]}' \
    -H 'Content-Type: application/json'

Using ChatHuggingFace in langchain this looks like the following:

llm_guard_chat = ChatHuggingFace(llm=llm_guard, model_id=safety_guard_model)
llm_guard_chat.invoke([{"role": "user", "content": input.text}]).content

If you do not add a model_id it will use the _resolve_model_id method to get the model_id but this does not work for locally deployed TGI services (because it uses huggingface_hub list_inference_endpoints function which only lists hugging face endpoints for account). This issue can be found here langchain-ai/langchain#17779. Current implementation errors as follows:

llm_engine_hf= ChatHuggingFace(llm=llm_guard)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain_community/chat_models/huggingface.py", line 54, in init
self._resolve_model_id()
File "/usr/local/lib/python3.11/site-packages/langchain_community/chat_models/huggingface.py", line 158, in _resolve_model_id
raise ValueError(
ValueError: Failed to resolve model_id Could not find model id for inference server provided: http://xx.xx.xx.xxx/
Make sure that your Hugging Face token has access to the endpoint.

This PR adds a workaround to this issue by adding a DEFAULT_MODEL variable and a way to try to get model id from /info endpoint. I also moved the ChatHuggingFace to run at start. This should improve latency of API calls by 1-2 seconds.

Issues

n/a.

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)

Dependencies

langchain-community

Tests

1. I rain two local TGI services of both LlamaGuard1 and

docker run -p 8087:80 -v $PWD/data:/data  -e HTTPS_PROXY=$https_proxy -e HTTP_PROXY=$https_proxy ghcr.io/huggingface/text-generation-inference --model-id meta-llama/LlamaGuard-7b

docker run -p 8087:80 -v $PWD/data:/data  -e HTTPS_PROXY=$https_proxy -e HTTP_PROXY=$https_proxy ghcr.io/huggingface/text-generation-inference --model-id meta-llama/Meta-Llama-Guard-2-8B

2. Then I build and ran guardrails-tgi-server

docker build -t opea/gen-ai-comps:guardrails-tgi-server --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/guardrails/langchain/docker/Dockerfile .

docker run -p 9090:9090  -e https_proxy=$https_proxy -e http_proxy=$http_proxy -e no_proxy=$no_proxy -e HUGGINGFACEHUB_API_TOKEN=$HUGGINGFACEHUB_API_TOKEN -e SAFETY_GUARD_ENDPOINT="http://localhost:8088" --network="host" opea/gen-ai-comps:guardrails-tgi-server

3. Tested with following curl:

$ curl http://localhost:9090/v1/guardrails -X POST   -d '{"text": "i am going to kill you"}'   -H 'Content-Type: application/json'
{"id":"e6a511430e2d5158d2923a4099502945","text":"Violated policies: Violent Crimes, please check your input."}twilbers@mlp-prod-skx-5675:/localdisk/twilbers/docker$

[tybrs]

@lvliang-intel Thanks for the review. Should we assign another person for review?

[tybrs]

@dcmiddle and @lvliang-intel Comments addressed and conflict fixed. Thanks!