From 2fe99d311bb661db911e4799a746bd2d66772530 Mon Sep 17 00:00:00 2001 From: teowu Date: Tue, 7 Nov 2023 15:39:59 +0800 Subject: [PATCH] add model zoo --- README.md | 76 +++++++++---------- .../.ipynb_checkpoints/README-checkpoint.md | 67 ++++++++++++++++ model_zoo/README.md | 67 ++++++++++++++++ 3 files changed, 169 insertions(+), 41 deletions(-) create mode 100644 model_zoo/.ipynb_checkpoints/README-checkpoint.md create mode 100644 model_zoo/README.md diff --git a/README.md b/README.md index 12086a8..4632ce5 100644 --- a/README.md +++ b/README.md @@ -47,10 +47,12 @@ ## Quick Start -If your server is facing poor connection to Huggingface, we provide an alternative way to [Download_Weights_from_ModelScope](use_with_modelscope/). Click in to see details. +If your server is facing poor connection to Huggingface, we provide an alternative way to [Download_Weights_from_ModelScope](/model_zoo#modelscope). Click in to see details. 对于中国大陆地区的使用者，若您的服务器连接huggingface存在一些困难，我们亦提供通过*魔搭*下载权重的方式。敬请点击参阅[指南](use_with_modelscope/). + + ### LLaVA-v1.5 #### Install LLaVA. @@ -134,36 +136,6 @@ python eval_scripts/llava_v1.5/eval_video_quality.py -#### Quantitative Evaluations - -
-Multi-choice question (MCQ) in Q-Bench. - -```shell -python eval_scripts/mplug_owl_2/eval_qbench_mcq.py -``` - -
- - -
-Image/Video Quality Assessment - -Image Quality Assessment: - -```shell -python eval_scripts/mplug_owl_2/eval_image_quality.py -``` - -Video Quality Assessment: - -```shell -python eval_scripts/mplug_owl_2/eval_video_quality.py -``` - -
- - ### mPLUG-Owl-2 *For mPLUG-Owl-2, Only Single GPU Inference is supported now. Please set environmental variable (e.g. `export CUDA_VISIBLE_DEVICES=0`) to make sure that the model can be loaded on only one device.* @@ -220,26 +192,48 @@ Note: The results may contain randomness as `do_sample=True` is enabled during c +#### Quantitative Evaluations -### InternLM-XComposer-VL +
+Multi-choice question (MCQ) in Q-Bench. + +```shell +python eval_scripts/mplug_owl_2/eval_qbench_mcq.py +``` -Coming soon. +
-## Model Zoo +
+Image/Video Quality Assessment -All weights are converted into Huggingface format and totally compatible with the base repositories ([LLaVA](https://github.com/haotian-liu/LLaVA/), [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl/), [InternLM-XComposer](https://github.com/InternLM/InternLM-XComposer)). After installing the base repositories, you can change the HF-path in the original evaluation scripts into the following ones, so as to automatically download the Q-Instruct-tuned versions. +Image Quality Assessment: + +```shell +python eval_scripts/mplug_owl_2/eval_image_quality.py +``` + +Video Quality Assessment: + +```shell +python eval_scripts/mplug_owl_2/eval_video_quality.py +``` -_Released_: +
+ + +### InternLM-XComposer-VL + +*---coming soon---* + + +## Model Zoo -- [LLaVA-v1.5-7B (mix)](https://huggingface.co/teowu/llava_v1.5_7b_qinstruct_preview_v0.1), HF-path: `teowu/llava_v1.5_7b_qinstruct_preview_v0.1` -- [LLaVA-v1.5-13B (mix)](https://huggingface.co/teowu/llava_v1.5_13b_qinstruct_preview_v0.1), HF-path: `teowu/llava_v1.5_13b_qinstruct_preview_v0.1` -- [mPLUG-Owl-2 (mix)](https://huggingface.co/teowu/mplug_owl2_7b_448_qinstruct_preview_v0.1), HF-path: `teowu/mplug_owl2_7b_448_qinstruct_preview_v0.1` -- [InternLM-XComposer-VL (mix)](https://huggingface.co/DLight/internlm_xcomposer_vl_qinstruct_preview_v0.1), HF-path: `DLight/internlm_xcomposer_vl_qinstruct_preview_v0.1` +See [Model Zoo](model_zoo). Both **huggingface** and **modelscope** weights are provided. ## Training -At present, we only provide the training scripts with LLaVA-v1.5. Please see [Training Docs](scripts/llava_v1.5) for more details. +At present, we only provide the training scripts with LLaVA-v1.5 (7B/13B). Please see [Training Docs](scripts/llava_v1.5) for more details. ## License diff --git a/model_zoo/.ipynb_checkpoints/README-checkpoint.md b/model_zoo/.ipynb_checkpoints/README-checkpoint.md new file mode 100644 index 0000000..b2e3696 --- /dev/null +++ b/model_zoo/.ipynb_checkpoints/README-checkpoint.md @@ -0,0 +1,67 @@ +## Model Zoo + + +### Huggingface + + +All weights are converted into Huggingface format and totally compatible with the base repositories ([LLaVA](https://github.com/haotian-liu/LLaVA/), [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl/), [InternLM-XComposer](https://github.com/InternLM/InternLM-XComposer)). See our [quick start](../README.md#quick-start) on how to use them. + +_Released_: + +- [LLaVA-v1.5-7B (mix)](https://huggingface.co/teowu/llava_v1.5_7b_qinstruct_preview_v0.1), HF-path: `teowu/llava_v1.5_7b_qinstruct_preview_v0.1` +- [LLaVA-v1.5-13B (mix)](https://huggingface.co/teowu/llava_v1.5_13b_qinstruct_preview_v0.1), HF-path: `teowu/llava_v1.5_13b_qinstruct_preview_v0.1` +- [mPLUG-Owl-2 (mix)](https://huggingface.co/teowu/mplug_owl2_7b_448_qinstruct_preview_v0.1), HF-path: `teowu/mplug_owl2_7b_448_qinstruct_preview_v0.1` +- [InternLM-XComposer-VL (mix)](https://huggingface.co/DLight1551/internlm-xcomposer-vl-7b-qinstruct-full), HF-path: `DLight1551/internlm-xcomposer-vl-7b-qinstruct-full` + + +### ModelScope + +If your server is facing poor connection to Huggingface, we provide an alternative way to download weights from ModelScope. Different from direct Huggingface weights, you need to use the two following steps to load them: + +#### Step 1: Download Weights + + +The links are as follows (WIP): + + +_Released_: + +- [LLaVA-v1.5-7B (mix)](https://www.modelscope.cn/models/qfuture/llava_v1.5_7b_qinstruct_preview_v0.1), HF-path: `qfuture/llava_v1.5_7b_qinstruct_preview_v0.1` +- [LLaVA-v1.5-13B (mix)](https://www.modelscope.cn/models/qfuture/llava_v1.5_13b_qinstruct_preview_v0.1), HF-path: `qfuture/llava_v1.5_13b_qinstruct_preview_v0.1` +- [mPLUG-Owl-2 (mix)](https://www.modelscope.cn/models/qfuture/mplug_owl_2_qinstruct_preview_v0.1), ModelScope-path: `qfuture/mplug_owl_2_qinstruct_preview_v0.1` + + +_Coming Soon_: + +- InternLM-XComposer-VL (mix) + +To use them, you need to install `Git LFS` and then clone the repositories directly from ModelScope, under the main directory of Q-Instruct. + +```shell +git clone https://www.modelscope.cn/models/qfuture/$MODEL_NAME_qinstruct_preview_v0.1.git +``` + +#### Step 2: Redirect the Model Paths to Your Local Directory + +After that, modify the `model_path` in [quick start](../README.md#quick-start) to the local path (i.e. `$MODEL_NAME_qinstruct_preview_v0.1`) to smoothly load the weights downloaded from ModelScope. + + +See the Example Code for Single Query on LLaVA-v1.5-7B below: + +```python +from llava.mm_utils import get_model_name_from_path +from llava.eval.run_llava import eval_model +model_path = "llava_v1.5_7b_qinstruct_preview_v0.1/" ## Modify Here to Your Local Relative Path ## +prompt = "Rate the quality of the image. Think step by step." +image_file = "fig/sausage.jpg" +args = type('Args', (), { + "model_path": model_path, + "model_base": None, + "model_name": get_model_name_from_path(model_path), + "query": prompt, + "conv_mode": None, + "image_file": image_file, + "sep": ",", +})() +eval_model(args) +``` diff --git a/model_zoo/README.md b/model_zoo/README.md new file mode 100644 index 0000000..b2e3696 --- /dev/null +++ b/model_zoo/README.md @@ -0,0 +1,67 @@ +## Model Zoo + + +### Huggingface + + +All weights are converted into Huggingface format and totally compatible with the base repositories ([LLaVA](https://github.com/haotian-liu/LLaVA/), [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl/), [InternLM-XComposer](https://github.com/InternLM/InternLM-XComposer)). See our [quick start](../README.md#quick-start) on how to use them. + +_Released_: + +- [LLaVA-v1.5-7B (mix)](https://huggingface.co/teowu/llava_v1.5_7b_qinstruct_preview_v0.1), HF-path: `teowu/llava_v1.5_7b_qinstruct_preview_v0.1` +- [LLaVA-v1.5-13B (mix)](https://huggingface.co/teowu/llava_v1.5_13b_qinstruct_preview_v0.1), HF-path: `teowu/llava_v1.5_13b_qinstruct_preview_v0.1` +- [mPLUG-Owl-2 (mix)](https://huggingface.co/teowu/mplug_owl2_7b_448_qinstruct_preview_v0.1), HF-path: `teowu/mplug_owl2_7b_448_qinstruct_preview_v0.1` +- [InternLM-XComposer-VL (mix)](https://huggingface.co/DLight1551/internlm-xcomposer-vl-7b-qinstruct-full), HF-path: `DLight1551/internlm-xcomposer-vl-7b-qinstruct-full` + + +### ModelScope + +If your server is facing poor connection to Huggingface, we provide an alternative way to download weights from ModelScope. Different from direct Huggingface weights, you need to use the two following steps to load them: + +#### Step 1: Download Weights + + +The links are as follows (WIP): + + +_Released_: + +- [LLaVA-v1.5-7B (mix)](https://www.modelscope.cn/models/qfuture/llava_v1.5_7b_qinstruct_preview_v0.1), HF-path: `qfuture/llava_v1.5_7b_qinstruct_preview_v0.1` +- [LLaVA-v1.5-13B (mix)](https://www.modelscope.cn/models/qfuture/llava_v1.5_13b_qinstruct_preview_v0.1), HF-path: `qfuture/llava_v1.5_13b_qinstruct_preview_v0.1` +- [mPLUG-Owl-2 (mix)](https://www.modelscope.cn/models/qfuture/mplug_owl_2_qinstruct_preview_v0.1), ModelScope-path: `qfuture/mplug_owl_2_qinstruct_preview_v0.1` + + +_Coming Soon_: + +- InternLM-XComposer-VL (mix) + +To use them, you need to install `Git LFS` and then clone the repositories directly from ModelScope, under the main directory of Q-Instruct. + +```shell +git clone https://www.modelscope.cn/models/qfuture/$MODEL_NAME_qinstruct_preview_v0.1.git +``` + +#### Step 2: Redirect the Model Paths to Your Local Directory + +After that, modify the `model_path` in [quick start](../README.md#quick-start) to the local path (i.e. `$MODEL_NAME_qinstruct_preview_v0.1`) to smoothly load the weights downloaded from ModelScope. + + +See the Example Code for Single Query on LLaVA-v1.5-7B below: + +```python +from llava.mm_utils import get_model_name_from_path +from llava.eval.run_llava import eval_model +model_path = "llava_v1.5_7b_qinstruct_preview_v0.1/" ## Modify Here to Your Local Relative Path ## +prompt = "Rate the quality of the image. Think step by step." +image_file = "fig/sausage.jpg" +args = type('Args', (), { + "model_path": model_path, + "model_base": None, + "model_name": get_model_name_from_path(model_path), + "query": prompt, + "conv_mode": None, + "image_file": image_file, + "sep": ",", +})() +eval_model(args) +```