Multimodal Embedding Microservice

[tileintel]

Description

This PR introduces multimodal embedding microservice using BridgeTower model as embedding model. This microservice is required for Multimodal RAG on Videos application

• We have added several dataclasses into comps/cores/proto/docarray.py that are required for the proposed microservices.
• We have provided a custom implementation of BridgeTower from the one on Huggingface, allowing to compute the embedding of text and the joint embedding of image-text pair.
• We have employed BridgeTower model for Multimodal Embedding Inference Endpoint (MMEI) running on both CPU and HPU.
• We have implemented multimodal embedding microservice with Local Multimodal Embedding Model (Local BridgeTower running on CPU)
• We have implemented multimodal embedding microservice with with MMEI_EMBEDDING_ENDPOINT.
• We have provided README file and tests.

Issues

RFC: https://github.com/opea-project/docs/pull/49/files
Issue: opea-project/GenAIExamples#358

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• [ x] New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

docarray[full]
fastapi
huggingface_hub
langchain
langsmith
opentelemetry-api
opentelemetry-exporter-otlp
opentelemetry-sdk
prometheus-fastapi-instrumentator
transformers
shortuuid
uvicorn
torch
torchvision
pydantic==2.8.2
BridgeTower

Tests

We have provided 2 tests for this microservice:

• tests/test_multimodal_embeddings_langchain_cpu.sh: This is to test microservice with MMEI running on CPU.
• tests/test_multimodal_embeddings_langchain_hpu.sh: This is to test microservice with MMEI running on CPU.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
comps/cores/proto/docarray.py	99.09% <100.00%> (+0.08%)	⬆️

... and 1 file with indirect coverage changes

[tileintel]

Hi @lvliang-intel. Thank you for your feedback about langsmith. We have removed langsmith from all places that you pointed out above. Would you please re-review this?

[XuhuiRen]

recommend to change the name of this folder. This title is to big to cover the multimodal functionality. Especially the use case for this code a pretty small domain where a text must paired with a image to obtain the embedding.

[tileintel]

Thanks for your feedback. We would like to discuss to keep the name of this folder as it is because of the followings:

1. Currently, it supports not only image-text pairs, but also support text as well. (c.f. please see method embed_query and embed_documents which inherit from interface Embedding from langchain
2. This BridgeTowerEmbedding can be extended to embed image (and embed other modalities) in future as well. In our current implementation, we haven't provided such methods because this will not be employed for our proposed application Multimodal RAG on Videos.
3. Although BridgeTowerEmbedding provides implemented methods for embed_documents, embed_query, embed_image_text_pairs, but it can also be considered as an interface for MultimodalEmbedding for other modalities for future development.
   We would appreciate if you can take into account these comments and consider again. Please let us know if any of these does not make sense and you insist in changing the folder name?
   Thanks a lot

[XuhuiRen]

as this PR is merged into 575, please close this PR

[tileintel]

This PR #555 is merged to the PR #575. Closing this one.
Thanks @XuhuiRen and @lvliang-intel