Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add video-llama LVM microservice under lvms #494

Closed
wants to merge 76 commits into from
Closed

Add video-llama LVM microservice under lvms #494

wants to merge 76 commits into from

Conversation

BaoHuiling
Copy link
Collaborator

Description

Add support for video-llama LVM, which allows us to chat based on video file with LVM. The first usecase is VideoRAGQnA use case from Multimodal RAG proposal.

Issues

List the issue or RFC link this PR is working on. If there is no such link, please mark it as n/a.
The RFC is still under review. opea-project/docs#49

Type of change

List the type of change like below. Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds new functionality)
  • Breaking change (fix or feature that would break existing design and interface)
  • Others (enhancement, documentation, validation, etc.)

Dependencies

python:3.9-slim git git-lfs faiss-cpu sentencepiece pandas Pillow prometheus_fastapi_instrumentator ftfy decord einops iopath omegaconf opencv-python-headless timm torch torchaudio langchain-core transformers webdataset pytorchvideo DAMO-NLP-SG/Video-LLaMA(github repo) DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned (huggingface repo)

Tests

Describe the tests that you ran to verify your changes.
https://github.com/siddhivelankar23/GenAIComps/blob/huiling-lvm/tests/test_lvms_video-llama.sh: build, start, validate, clean the microservice

@BaoHuiling BaoHuiling force-pushed the huiling-lvm branch 2 times, most recently from 461cfa8 to 0574132 Compare August 15, 2024 10:16
Spycsh and others added 22 commits August 15, 2024 18:25
@Spycsh @BaoHuiling
fix wrong indent (opea-project#364)
bf8c09e 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
initial commit
8d6427b 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
update video llama, model download
3117dd1 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
add test script
467d91b 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
Add LVM container, update video_path to URL
ec278d9 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
update for prod
f3a79ad 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
save
ee18b10 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
test script passed
565d6bf 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
add readme
f71ea37 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
add clean in readme
5ff6630 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
change input param name to "input"
a1f1257 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
fix typo
2856073 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
Add support for UI
4a4de0c 
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
fix typo in test
2b35c01 
Signed-off-by: BaoHuiling <[email protected]>
@bjzhjing @Spycsh @BaoHuiling
Redis-dataprep: Make Redis connection consistent (opea-project#359)
6875d48 
There are both 'from langchain_community.vectorstores import Redis' and
'import redis' in prepare_doc_redis.py. Redis.from_texts_return_keys()
is with REDIS_URL as connection, while redis.Redis() takes the
combination of REDIS_HOST and REDIS_PORT as connection. When the former
and the latter are not the same, it will fail to get data from vector
database. Change to use REDIS_URL as the unified way.

Signed-off-by: Cathy Zhang <[email protected]>
Co-authored-by: Sihan Chen <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@letonghan @pre-commit-ci @BaoHuiling
Update Dataprep with Parameter Settings (opea-project#351)
cf4a43b 
* update dataprep with parameter settings

Signed-off-by: letonghan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update port

Signed-off-by: letonghan <[email protected]>

---------

Signed-off-by: letonghan <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <[email protected]>
@letonghan @pre-commit-ci @BaoHuiling
Support delete for Milvus vector db in Dataprep (opea-project#368)
8fe43ea 
* support delete for milvus

Signed-off-by: letonghan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <[email protected]>
@ZePan110 @BaoHuiling
Fix dockerfile format (opea-project#363)
82a3d10 
Signed-off-by: zepan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@chensuyue @pre-commit-ci @BaoHuiling
refactor build image build and push workflow (opea-project#385)
c3698b0 
* refactor build image build and push workflow

Signed-off-by: chensuyue <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <[email protected]>
@minmin-intel @pre-commit-ci
@chensuyue @BaoHuiling
Add ML detection strategy to PII detection guardrail (opea-project#292)
7df37e0 
* first code of business safety classifier

* allow strategy options and update readme

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* disable ray for ml strategy and update test script

* add log to pii test

Signed-off-by: minmin-intel <[email protected]>

* update logging in test gaurdrail

* rm llm strategy and change url in test

Signed-off-by: minmin-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* delete file check in test and update readme

Signed-off-by: minmin-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: minmin-intel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@letonghan @pre-commit-ci @BaoHuiling
Refine Dataprep Code & UT (opea-project#404)
60d2850 
* refine dataprep upload_link & ut

Signed-off-by: letonghan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add requirement

Signed-off-by: letonghan <[email protected]>

* modify container name

Signed-off-by: letonghan <[email protected]>

---------

Signed-off-by: letonghan <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <[email protected]>
@lvliang-intel @BaoHuiling
Add Uvicorn dependency (opea-project#408)
ab51e7e 
Signed-off-by: lvliang-intel <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
hteeyeoh and others added 29 commits August 15, 2024 18:28
@hteeyeoh @pre-commit-ci @BaoHuiling
prompt_registry: Unifying API endpoint port (opea-project#443)
7ae1b85 
* prompt_registry: Unifying API endpoint port

Unifying all port for API endpoints for prompt registry services with
recent register_microservice wrapper function update.

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Yeoh, Hoong Tee <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <[email protected]>
@Spycsh @BaoHuiling
fix ffmpeg build on hpu (opea-project#441)
1dbcf83 
Signed-off-by: BaoHuiling <[email protected]>
@ZePan110 @BaoHuiling
Add path check for dockerfiles in compose.yaml and change workflow na…
5830f07 
…me. (opea-project#412)

Signed-off-by: zepan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@zahidulhaque @BaoHuiling
Fix OpenVINO vLLM build scripts and update unit test case (opea-proje…
74d6476 
…ct#446)

Signed-off-by: Zahidul Haque <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@chensuyue @BaoHuiling
Optional container build instructions (opea-project#457)
a497345 
Signed-off-by: chensuyue <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@xiguiw @BaoHuiling
LLM micro service extract input model name (opea-project#447)
cc5b741 
No input model from UI.
vLLM and OLLAMA get the model name from the envrionment
TGI mode model form ModelID

Signed-off-by: Wang, Xigui <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@lkk12014402 @sys-lpot-val
@lvliang-intel @BaoHuiling
refact embedding/ranking/llm request/response by referring to openai …
dacb69b 
…format (opea-project#405)

Co-authored-by: sys-lpot-val <[email protected]>
Co-authored-by: lvliang-intel <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@pallavijaini0525 @BaoHuiling
Pinecone update to OPEA (opea-project#420)
ca545cf 
Signed-off-by: pallavi jaini <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@rbrugaro @BaoHuiling
remove deprecated langchain imports and switch to langchain-huggingfa…
7546b23 
…ce (opea-project#417)

* remrove deprecated imports and switch to langchain-huggingface

Signed-off-by: rbrugaro <[email protected]>

* fix typo

Signed-off-by: rbrugaro <[email protected]>

* set requirements to langchain latest release

Signed-off-by: rbrugaro <[email protected]>

* remove version for langchain in requirements

Signed-off-by: rbrugaro <[email protected]>

---------

Signed-off-by: rbrugaro <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@yogeshmpandey @BaoHuiling
Added support for Unified Port, GET/DELETE endpoints in pgvector (ope…
ad8317f 
…a-project#444)

Signed-off-by: Yogesh <[email protected]>
Co-authored-by: Yogesh <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@ZePan110 @pre-commit-ci @BaoHuiling
Fix validation failure without exit. (opea-project#460)
aac314a 
* Fix validation failure without exit.

Signed-off-by: zepan <[email protected]>

* Change validate log.

Signed-off-by: zepan <[email protected]>

* Fix validation failure without exit.

Signed-off-by: zepan <[email protected]>

* Change validate log.

Signed-off-by: zepan <[email protected]>

* Change validate log.

Signed-off-by: zepan <[email protected]>

* Fix issue for reranking-langchain-mosec.

Signed-off-by: zepan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix script issue

Signed-off-by: zepan <[email protected]>

---------

Signed-off-by: zepan <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <[email protected]>
@lianhao @BaoHuiling
whisper: Fix container build failure (opea-project#470)
c354a04 
Signed-off-by: Lianhao Lu <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@letonghan @BaoHuiling
Update Milvus docker-compose.yaml (opea-project#459)
e72ec0f 
Signed-off-by: letonghan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@letonghan @BaoHuiling
Refine robustness of Dataprep Redis (opea-project#463)
c8e347a 
Signed-off-by: letonghan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@chensuyue @BaoHuiling
clean local images before test (opea-project#471)
446aff0 
* clean test images before test

Signed-off-by: chensuyue <[email protected]>

* add --no-cache for image build

Signed-off-by: chensuyue <[email protected]>

---------

Signed-off-by: chensuyue <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@ZePan110 @BaoHuiling
Fix embeddings_langchain-mosec issue. (opea-project#473)
3262aee 
Signed-off-by: zepan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@ganesanintel @BaoHuiling
Address test case failure (opea-project#462)
62360c1 
Signed-off-by: V, Ganesan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@letonghan @BaoHuiling
Fix Dataprep qdrant issues and add Test Script (opea-project#474)
1e342c4 
Signed-off-by: letonghan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@wangkl2 @BaoHuiling
Refine the instructions to run the retriever example with qdrant (ope…
c7570ec 
…a-project#442)

Signed-off-by: Wang, Kai Lawrence <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@rbrugaro @pre-commit-ci @BaoHuiling
fix HuggingFaceEmbedding deprecated in favor of HuggingFaceInferenceA…
1fcc547 
…PIEmbedding (opea-project#466)

* Fix HuggingFaceEmbedding deprecated in favor of HuggingFaceInferenceAPIEmbedding

Signed-off-by: rbrugaro <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: rbrugaro <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <[email protected]>
@ZePan110 @BaoHuiling
update test scripts and bug fix (opea-project#464)
1acda9e 
Signed-off-by: chensuyue <[email protected]>
Signed-off-by: zepan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling
Add output evaluation for guardrails (opea-project#332)
2a38ac4 
* add single query input/output guardrails

Signed-off-by: Tyler Wilbers <[email protected]>

* removed comment

Signed-off-by: Tyler Wilbers <[email protected]>

---------

Signed-off-by: Tyler Wilbers <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@lvliang-intel @BaoHuiling
Revert max_new_tokens to 1024 (opea-project#479)
e11d0c9 
Signed-off-by: lvliang-intel <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@letonghan @BaoHuiling
fix empty content issue (opea-project#476)
0b647c8 
Signed-off-by: letonghan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@Spycsh @pre-commit-ci
@chensuyue @BaoHuiling
support whisper long-form generation (opea-project#469)
e59594b 
* fix long form asr accuracy

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix test input pad issue

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@ZePan110 @BaoHuiling
Fix issue (opea-project#483)
b749da8 
* Standardize image names and labels.

Signed-off-by: zepan <[email protected]>

* Remove test cases that are no longer in use.

Signed-off-by: zepan <[email protected]>

---------

Signed-off-by: zepan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@ZePan110 @BaoHuiling
Change /root/ to /home/user/. (opea-project#475)
0862d2b 
* Change /root/ to /hmoe/user/.

Signed-off-by: zepan <[email protected]>

* Fix issue.

Signed-off-by: zepan <[email protected]>

---------

Signed-off-by: zepan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@ZePan110 @BaoHuiling
Check build fail and change port 8008 to 5025/5026. (opea-project#487)
91278e0 
Signed-off-by: zepan <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@XuehaoSun @BaoHuiling
Freeze requirements workflow for release tag (opea-project#485)
0574132 
Signed-off-by: Sun, Xuehao <[email protected]>
Signed-off-by: NeuralChatBot <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
@BaoHuiling BaoHuiling closed this Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lvliang-intel lvliang-intel Awaiting requested review from lvliang-intel lvliang-intel is a code owner

@Spycsh Spycsh Awaiting requested review from Spycsh

@XuhuiRen XuhuiRen Awaiting requested review from XuhuiRen

@letonghan letonghan Awaiting requested review from letonghan

@XinyuYe-Intel XinyuYe-Intel Awaiting requested review from XinyuYe-Intel

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.