Introduction

This is a PyTorch implementation for the AAAI 2024 paper "Negative Pre-aware for Noisy Cross-Modal Matching". Our method NPC is built on top of the CLIP in PyTorch for end-to-end Image-text Matching.

In Step 1, we calculate the negative impact of each sample via the siamese model A' of base model A.

In Step 2, we train the base model A with the re-weight samples and memory bank.

The proposed NPC achieves much better accuracy (higher R@1) and higher robustness (lower variance among R@1).

Requirements

• python 3.7
• numpy 1.21.6 (> 1.19.5)
• torch 1.13.1
• tensorboard 2.11.2

pip install requirments.txt

Data Preparation

Split Dataset

We conducted experiments on three datasets: MSCOCO, Flickr30K, and CC120K. We followed SCAN to split image-text pairs in MSCOCO and FLickr30K into training, validation and testing sets.

• MSCOCO. We unified the images' name format of the MSCOCO dataset for easier use. You can use dataset/MSCOCO_rename.py to rename the images in MSCOCO. (MSCOCO_2014)
• Flickr30K. (Flickr30K).
• CC120K. We tested the proposed method on the real-world dataset Conceptual Captions. Since the full dataset is too large, we randomly selected a subset of Conceptual Captions, named CC120K, including 118,851 images for training, 1,000 for validation, and 1,000 for testing. You can download the dataset from here with the code "3ble".

Construct Noisy Datasets

We constructed noise by randomly shuffling some captions of the images.

You can obtain your noisy dataset using construct_noise.py. And we also integrated this part of the code in data.py. The noisy dataset will be automatically constructed and saved during training when it doesn't exist.

Since there are around 3%-20% incorrect annotations existing in the real-world dataset Conceptual Captions, we did not create noisy samples manually.

Frozen Memory Bank

We provide the frozen Memory Bank that appeared in the paper for the datasets with different noise ratios. If you want to update them during training, please use get_memorybank.py.

Note!

If you want to use your own noisy dataset for training, the Memory Bank should also be rebuilt. You can construct the noise dataset by construct_noise.py, and obtain the Memory Bank by get_memorybank.py.

Download Link

• MSCOCO
• Flickr30K
• CC120K

The final data directory tree should be:

(Fix a mistake: the path has been defined as dataset/${DATASET_NAME}/annotations/frozen_memory_bank before, and the correct one should be dataset/${DATASET_NAME}/annotations/memory_bank. Thanks for correction！)

├── dataset/
├── ${DATASET_NAME}/
|    ├── annotations/
|    |   ├── memory_bank/
|    |   |   ├── ${noise_ratio}_mbank_img_idx.npy
|    |   |   ├── ${noise_ratio}_mbank_txt_idx.npy
|    |   |   └── ...
|    |   └──scan_split/
|    |       ├── ${noise_ratio}_noise_train_caps.txt #samples use for training. ${noise_ration} is in {0, 0.2, 0.4, 0.6}
|    |       ├── train_caps.txt # the same as `0_noise_train_caps.txt`
|    |       ├── train_ids.txt 
|    |       ├── dev_caps.txt #samples use for validation
|    |       ├── dev_ids.txt 
|    |       ├── test_caps.txt #samples use for testing
|    |       ├── test_ids.txt 
|    |       └── ...
|    └── images/ # all images in MSCOCO (or Flickr30K, CC120K)
└── ...

Models and Evaluation

You can download the models fine-tuned using NPC(ours) and CLIP(our baseline) from this link.

Save the models in folder ./pre-trained_models, and evaluate the models via the following command. For example, evaluate the models trained on MSCOCO with 60% noise.

python main_NPC.py --eval --resume /AAAI24-NPC/pre-trained_models/npc_coco_60.pt --dataset_root /AAAI24-NPC/dataset/MSCOCO --dataset coco

python main_CLIP.py --eval --resume /AAAI24-NPC/pre-trained_models/clip_coco_60.pt --dataset_root /AAAI24-NPC/dataset/MSCOCO --dataset coco

• Experiment results on MSCOCO 1K and 5K.

Noise Ratio	Method	MSCOCO 1K						MSCOCO 5K
		Image-to-Text			Text-to-Image			Image-to-Text			Text-to-Image
		R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
0%	CLIP	79.9	95.1	98.1	65.0	90.3	98.1	62.2	84.6	90.9	45.1	72.3	81.8
	NPC	82.2	96.5	98.7	68.3	92.0	98.7	65.4	87.3	93.1	48.5	75.4	84.4
20%	CLIP	75.0	93.1	97.2	58.7	86.1	97.2	55.5	80.3	87.8	38.8	65.5	75.7
	NPC	79.9	95.9	98.4	66.3	90.8	98.4	61.6	85.4	91.6	46.0	73.4	82.9
40%	CLIP	70.7	91.7	96.2	54.7	83.4	96.2	51.6	76.1	84.7	35.3	61.3	72.1
	NPC	79.4	95.1	98.3	65.0	90.1	98.3	61.3	84.7	90.6	44.7	72.2	81.7
50%	CLIP	69.0	90.6	95.8	52.8	82.0	95.8	49.0	74.5	83.0	33.7	59.1	70.1
	NPC	78.5	95.0	98.1	64.1	89.6	98.1	60.1	83.5	90.4	43.6	71.1	80.9
60%	CLIP	67.0	88.8	95.0	49.7	79.6	95.0	45.1	71.5	80.9	30.9	55.6	66.9
	NPC	78.2	94.4	97.7	63.1	89.0	97.7	59.9	82.9	89.7	43.0	70.2	80.0

• Experiment results on Flickr30K.

Noise Ratio	Method	Image-to-Text			Text-to-Image
		R@1	R@5	R@10	R@1	R@5	R@10
0%	CLIP	86.2	97.6	99.2	72.9	92.3	96.0
	NPC	87.9	98.1	99.4	75.0	93.7	97.2
20%	CLIP	82.3	95.5	98.3	66.0	88.5	93.5
	NPC	87.3	97.5	98.8	72.9	92.1	95.8
40%	CLIP	76.2	93.3	96.5	59.4	85.0	90.9
	NPC	85.6	97.5	98.4	71.3	91.3	95.3
60%	CLIP	66.3	87.3	93.0	52.1	78.8	87.4
	NPC	83.0	95.9	98.6	68.1	89.6	94.2

• Experiment results on CC120K.

Method	Image-to-Text			Text-to-Image
	R@1	R@5	R@10	R@1	R@5	R@10
CLIP	68.8	87.0	92.9	67.8	86.4	90.9
NPC	71.1	92.0	96.2	73.0	90.5	94.8

Training

For training NPC You can train a new model via the following command. Before training, you can read the params.py carefully to check your parameter setting. The --num_anns should be set to 5 for MSCOCO and Flickr30K, and 1 for CC120K.

python main_NPC.py --batch_size 256 --epochs 5 --lr 2e-7 --vision_model ViT-B/32 --noise_ratio ${NOISE RATIO} --num_anns ${5 or 1} --dataset_root ${YOUR PATH} --dataset coco --checkpoint_path ${YOUR PATH}

For training CLIP Thanks to this project for providing a basic fine-tuning framework of CLIP. We have improved the code of the data loading process and the model evaluation. The --num_anns should be set to 5 for MSCOCO and Flickr30K, and 1 for CC120K. You can fine-tune the CLIP via the following command.

python main_CLIP.py --batch_size 256 --epochs 5 --lr 5e-7 --vision_model ViT-B/32 --noise_ratio ${NOISE RATIO} --num_anns ${5 or 1} --dataset_root ${YOUR PATH} --dataset coco --checkpoint_path ${YOUR PATH}

Reference

If our proposed NPC is helpful for you, welcome to cite the following paper. 😃

@inproceedings{NPC,
author = {Xu Zhang and Hao Li and Mang Ye},
title = {Negative Pre-aware for Noisy Cross-Modal Matching},
booktitle = {AAAI},
year = {2024}
}