Add output evaluation for guardrails

[tybrs]

Description

This PR attempts to reduce latency by allowing safeguard microservice to be at end of DAG and process safety of both input and output with single query. This means you can structure the DAG as follows to reduce latency of RAG flow:

        self.megaservice.add(embedding).add(retriever).add(rerank).add(llm).add(guardrails)
        self.megaservice.flow_to(embedding, retriever)
        self.megaservice.flow_to(retriever, rerank)
        self.megaservice.flow_to(rerank, llm)
        self.megaservice.flow_to(llm, guardrails)
        self.gateway = ChatQnAGateway(megaservice=self.megaservice, host="0.0.0.0", port=self.port)

Currently our implementation only supports single input safeguarding. But the chat Messages API (v1/chat/completion) will allow for templating for both "user" inputs and "assistant" output. with a single query. This PR adds the ability to send a list of both "user" and "assistant" messages for safeguarding. Since LLM outputs both "prompt and "text" with GeneratedDoc this means you can feed output directly into guardrails microservice.

curl http://localhost:9090/v1/guardrails \
  -X POST \
  -d '{
    "prompt" : "How do you buy a tiger in the US",
    "text" : "Yes! Buy a tiger in the US.",
    "parameters":{"max_new_tokens":32}
  }

However it maintains backwards compatibility with following DAG:

        self.megaservice.add(guardrail_in).add(embedding).add(retriever).add(rerank).add(llm).add(guardrail_out)
        self.megaservice.flow_to(guardrail_in, embedding)
        self.megaservice.flow_to(embedding, retriever)
        self.megaservice.flow_to(retriever, rerank)
        self.megaservice.flow_to(rerank, llm)
        self.megaservice.flow_to(llm, guardrail_out)
        self.gateway = ChatQnAGateway(megaservice=self.megaservice, host="0.0.0.0", port=self.port)

It also can be queried with just text:

curl http://localhost:9090/v1/guardrails \
  -X POST \
  -d '{
    "text" : "How do you buy a tiger in the US",
    "parameters":{"max_new_tokens":32}
  }

Issues

n/a.

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

Changed to hugingface_hub<=0.24.0. because HugingFaceEndpoint does it assigns both endpoint_url or repo_id to InferenceClient.model attribute. But this is a bug since hugingface_hub>0.24.0. introduces a base_url kwarg for InferenceClient to be used.

Tests

Ran the following

docker run -d --name="guardrails-tgi-server" -p 9090:9090 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e no_proxy=$no_proxy -e SAFETY_GUARD_ENDPOINT=${SAFETY_GUARD_ENDPOINT} -e HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN} opea/guardrails-tgi:latest

curl http://localhost:9090/v1/guardrails   -X POST   -d '{
    "messages":[{"role": "user", "text" : "How do you buy a tiger in the US?"}],
    "parameters":{"max_new_tokens":32}
  }'   -H 'Content-Type: application/json'

Output:

{"downstream_black_list":[".*"],"id":"6e70dcfa9db13087fc390d84c7869e7a","text":"Violated policies: Violent Crimes, please check your input."}

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files	Coverage Δ
comps/cores/proto/docarray.py	100.00% <100.00%> (ø)

[tybrs]

@lvliang-intel @letonghan This PR should be ready for review. Is there a benchmark for latency with guardrails? I would love to measure the potential effect.

[ashahba]

LGTM!

[ashahba]

LGTM!