Why group_by_modality is set to False on the custom_finetune.sh script?

[ggcr]

Hi,

I am exploring on doing different experiments, by training the projector only, and I've seen that the default provided script for custom finetuning has the group_by_modality_length flag set to False.

TinyLLaVA_Factory/scripts/train/custom_finetune.sh

Lines 1 to 24 in f5c2ded

	DATA_PATH="/home/ai/data/llava/dataset/text_files/llava_v1_5_mix665k.json"
	IMAGE_PATH="/home/ai/data/llava/dataset"
	MODEL_MAX_LENGTH=3072
	OUTPUT_DIR="/mnt/data/sata/yinghu/checkpoints/llava_factory/custom-finetune-TinyLLaVA-Phi-2-SigLIP-3.1B-lora"
	deepspeed --include localhost:0,1,2,3 --master_port 29501 tinyllava/train/custom_finetune.py \
	--deepspeed ./scripts/zero2.json \
	--data_path $DATA_PATH \
	--image_folder $IMAGE_PATH \
	--is_multimodal True \
	--conv_version phi \
	--mm_vision_select_layer -2 \
	--image_aspect_ratio square \
	--fp16 True \
	--training_recipe lora \
	--tune_type_llm lora \
	--tune_type_vision_tower frozen \
	--tune_vision_tower_from_layer 0 \
	--tune_type_connector full \
	--lora_r 128 \
	--lora_alpha 256 \
	--group_by_modality_length False \
	--pretrained_model_path "tinyllava/TinyLLaVA-Phi-2-SigLIP-3.1B" \
	--output_dir $OUTPUT_DIR \

Because it has the llava_v1_5_mix665k.json as Data Path, it is not a uni-modal dataset, as it contains conversations with the <image> mm token.

Can you clarify when we should set the group_by_modality_length flag to True?

I am running in errors that are due to this:

E.g.

File "/root/TinyLLaVA_Factory/tinyllava/train/tinyllava_trainer.py", line 47, in get_modality_length_grouped_indices
    lang_indices, lang_lengths = zip(*[(i, -l) for i, l in enumerate(lengths) if l < 0])
ValueError: not enough values to unpack (expected 2, got 0)

Thanks in advance.

[ggcr]

There is an inconsistency, in some (full) finetuning scripts it is set to False, and on others True.

[ggcr]

I have a dataset of pure multi-modal data, that is, each question ("from": "gpt") contains the <image> label. Hence, this error should not pop up.

LLaVA author addresses it in this issue.

I created a PR with the same fix that the main LLaVA repo currently is using.

[ZhangXJ199]

Thanks for your question, group_by_modality_length needs to be set according to the data type: for data that only contains multimodal QA, group_by_modality_length needs to be set to False; for data that contains both multimodal QA and pure text QA (such as SQA), group_by_modality_length can be set to True.