adding embedding support for CLIP based models for VideoRAGQnA example for v0.9 #538

srinarayan-srikanthan · 2024-08-21T04:20:45Z

Description

Add support for CLIP based embeddings for multimodal VideoRAG usecase.

Issues

List the issue or RFC link this PR is working on. If there is no such link, please mark it as n/a.
The RFC is under review : opea-project/docs#49

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

List the newly introduced 3rd party dependency if exists.

Tests

Describe the tests that you ran to verify your changes.

srinarayan-srikanthan · 2024-08-21T04:21:12Z

@chensuyue can you please tag this PR with 'V0.9' Milestone

codecov · 2024-08-21T04:24:24Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
comps/cores/proto/docarray.py	`99.12% <100.00%> (+<0.01%)`	⬆️

BaoHuiling · 2024-08-22T01:52:18Z

@XuhuiRen @lvliang-intel Could you help to review the code? Thanks

comps/embeddings/langchain_multimodal/README.md

comps/embeddings/langchain_multimodal/embedding_multimodal.py

comps/embeddings/langchain_multimodal/requirements.txt

lvliang-intel · 2024-08-29T01:16:23Z

@srinarayan-srikanthan,
Please also check why the example test failed.

ttrigui · 2024-08-29T23:51:33Z

@lvliang-intel , the example test fail seem to be unrelated to the PR changes. This test seems to fail in another PR too.

XuhuiRen · 2024-08-30T02:01:51Z

please fix the name confliction with #575 before merging

BaoHuiling · 2024-08-30T03:58:05Z

For the failed test "Example-test", here is the comment from @chensuyue

The test failed at megaservice step due to mismatch of the expected keyword and output from curl

To fix this, please do local test on your side with this script to see what happens. Please note you need to modify line 19 to checkout to your branch under testing. (for example, git clone -b sri-clip-embedding https://github.com/siddhivelankar23/GenAIComps.git)

Signed-off-by: srinarayan-srikanthan <[email protected]>

for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: srinarayan-srikanthan <[email protected]>

for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: Sun, Xuehao <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

* fix get file error & refine logs Signed-off-by: letonghan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: letonghan <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]>

…oject#541) Signed-off-by: letonghan <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: srinarayan-srikanthan <[email protected]>

…pea-project#313) Signed-off-by: Chendi.Xue <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: XuhuiRen <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: Xinyu Ye <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: lkk <[email protected]> Co-authored-by: test <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Letong Han <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: chensuyue <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

* Use Gaudi base images from Dockerhub Signed-off-by: Abolfazl Shahbazi <[email protected]> * Fixing the malformed tag Signed-off-by: Abolfazl Shahbazi <[email protected]> * fix another malformed tag Signed-off-by: Abolfazl Shahbazi <[email protected]> --------- Signed-off-by: Abolfazl Shahbazi <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add toxicity detection microservice Signed-off-by: Qun Gao <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Modification to toxicity plugin PR (opea-project#432) * changed microservice to use Service.GUARDRAILS and input/output to TextDoc Signed-off-by: Tyler Wilbers <[email protected]> * simplify dockerfile to use langchain Signed-off-by: Tyler Wilbers <[email protected]> * sort requirements Signed-off-by: Tyler Wilbers <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Tyler Wilbers <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Minor SPDX header update (opea-project#434) Signed-off-by: Abolfazl Shahbazi <[email protected]> * Remove 'langsmith' per code review (opea-project#534) Signed-off-by: Abolfazl Shahbazi <[email protected]> * Add toxicity detection microservices with E2E testing Signed-off-by: Qun Gao <[email protected]> --------- Signed-off-by: Qun Gao <[email protected]> Signed-off-by: Tyler Wilbers <[email protected]> Signed-off-by: Abolfazl Shahbazi <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abolfazl Shahbazi <[email protected]> Co-authored-by: Tyler W <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: srinarayan-srikanthan <[email protected]>

for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]>

for more information, see https://pre-commit.ci

XuhuiRen · 2024-09-03T05:03:19Z

@BaoHuiling @srinarayan-srikanthan please consider to have a better name to distinguish your previous PR for the folder https://github.com/opea-project/GenAIComps/tree/main/comps/embeddings/multimodal_embeddings/multimodal_langchain and the folder "comps/embeddings/langchain_multimodal/" in this PR. These two names are too similar. The name should have a clarify representation. Otherwise, it will lead to unexpected confusion.

Signed-off-by: BaoHuiling <[email protected]>

ttrigui · 2024-09-03T21:15:37Z

@lvliang-intel , can we please merge this PR since all issues are resolved?

Signed-off-by: srinarayan-srikanthan <[email protected]>

srinarayan-srikanthan requested review from XuhuiRen and lvliang-intel as code owners August 21, 2024 04:20

BaoHuiling mentioned this pull request Aug 21, 2024

adding embedding support for CLIP based models for VideoRAGQnA #533

Closed

1 task

srinarayan-srikanthan changed the title ~~adding embedding support for CLIP based models for VideoRAGQnA example~~ adding embedding support for CLIP based models for VideoRAGQnA example for v0.9 Aug 21, 2024

tileintel mentioned this pull request Aug 21, 2024

Multimodality opea-project/GenAIExamples#358

Closed

lvliang-intel reviewed Aug 29, 2024

View reviewed changes

comps/embeddings/langchain_multimodal/README.md Outdated Show resolved Hide resolved

lvliang-intel reviewed Aug 29, 2024

View reviewed changes

comps/embeddings/langchain_multimodal/embedding_multimodal.py Outdated Show resolved Hide resolved

lvliang-intel reviewed Aug 29, 2024

View reviewed changes

comps/embeddings/langchain_multimodal/embedding_multimodal.py Outdated Show resolved Hide resolved

lvliang-intel reviewed Aug 29, 2024

View reviewed changes

comps/embeddings/langchain_multimodal/requirements.txt Outdated Show resolved Hide resolved

BaoHuiling added this to the v1.0 milestone Aug 29, 2024

XuhuiRen approved these changes Aug 29, 2024

View reviewed changes

BaoHuiling mentioned this pull request Aug 29, 2024

Multimodal dataprep #575

Merged

3 tasks

BaoHuiling mentioned this pull request Aug 30, 2024

add VDMS retriever microservice for v0.9 Milestone #539

Merged

srinarayan-srikanthan and others added 10 commits September 1, 2024 10:36

clip embedding support

cb5b2bb

Signed-off-by: srinarayan-srikanthan <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

885af46

for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]>

test script for embedding

c793514

Signed-off-by: srinarayan-srikanthan <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

d9efb3a

for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]>

fix freeze workflow (opea-project#522)

6926c15

Signed-off-by: Sun, Xuehao <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

Support SearchedDoc input type in LLM for No Rerank Pipeline (opea-pr…

4127ff6

…oject#541) Signed-off-by: letonghan <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

Add dependency for pdf2image and OCR processing (opea-project#421)

7e660a1

Signed-off-by: srinarayan-srikanthan <[email protected]>

Add local_embedding return 768 length to align with chatqna example (o…

f75267d

…pea-project#313) Signed-off-by: Chendi.Xue <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

add telemetry doc (opea-project#536)

389fb61

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]>

BaoHuiling and others added 12 commits September 1, 2024 10:36

Add video-llama LVM microservice under lvms (opea-project#495)

bb01c19

Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]>

Fix the data load issue for structured files (opea-project#505)

7a7f377

Signed-off-by: XuhuiRen <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

add torchvision into requirements (opea-project#546)

8ca6484

Signed-off-by: chensuyue <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

rename script and use 5xxx

41cc41f

Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

add proxy for build

64c984b

Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]>

fixed commit issues

a3bb34c

Signed-off-by: srinarayan-srikanthan <[email protected]>

Fix docarray constraint

b72d2ba

Signed-off-by: srinarayan-srikanthan <[email protected]>

updated docarray

36620a8

Signed-off-by: srinarayan-srikanthan <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

532741d

for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]>

srinarayan-srikanthan force-pushed the sri-clip-embedding branch from 1d26b57 to 532741d Compare September 1, 2024 17:39

srinarayan-srikanthan requested review from Spycsh, letonghan and XinyuYe-Intel as code owners September 1, 2024 17:39

srinarayan-srikanthan and others added 2 commits September 2, 2024 12:42

Merge branch 'main' into sri-clip-embedding

5dc564b

[pre-commit.ci] auto fixes from pre-commit.com hooks

cdda846

for more information, see https://pre-commit.ci

rm telemetry which cause error in mega

bc318c0

Signed-off-by: BaoHuiling <[email protected]>

srinarayan-srikanthan and others added 3 commits September 3, 2024 14:44

renamed dirs

6289315

Signed-off-by: srinarayan-srikanthan <[email protected]>

renamed test

9878ba0

Signed-off-by: srinarayan-srikanthan <[email protected]>

Merge branch 'main' into sri-clip-embedding

58d83cd

yinghu5 self-requested a review September 4, 2024 00:54

yinghu5 approved these changes Sep 4, 2024

View reviewed changes

yinghu5 merged commit 2a53e25 into opea-project:main Sep 4, 2024
19 checks passed

lkk12014402 pushed a commit that referenced this pull request Sep 19, 2024

fix script issues in MD file (#538)

acdd712

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding embedding support for CLIP based models for VideoRAGQnA example for v0.9 #538

adding embedding support for CLIP based models for VideoRAGQnA example for v0.9 #538

srinarayan-srikanthan commented Aug 21, 2024

srinarayan-srikanthan commented Aug 21, 2024

codecov bot commented Aug 21, 2024 •

edited

Loading

BaoHuiling commented Aug 22, 2024

lvliang-intel commented Aug 29, 2024

ttrigui commented Aug 29, 2024

XuhuiRen commented Aug 30, 2024

BaoHuiling commented Aug 30, 2024

XuhuiRen commented Sep 3, 2024

ttrigui commented Sep 3, 2024

adding embedding support for CLIP based models for VideoRAGQnA example for v0.9 #538

adding embedding support for CLIP based models for VideoRAGQnA example for v0.9 #538

Conversation

srinarayan-srikanthan commented Aug 21, 2024

Description

Issues

Type of change

Dependencies

Tests

srinarayan-srikanthan commented Aug 21, 2024

codecov bot commented Aug 21, 2024 • edited Loading

Codecov Report

BaoHuiling commented Aug 22, 2024

lvliang-intel commented Aug 29, 2024

ttrigui commented Aug 29, 2024

XuhuiRen commented Aug 30, 2024

BaoHuiling commented Aug 30, 2024

XuhuiRen commented Sep 3, 2024

ttrigui commented Sep 3, 2024

codecov bot commented Aug 21, 2024 •

edited

Loading