Skip to content

Latest commit

 

History

History
103 lines (81 loc) · 4.59 KB

README.md

File metadata and controls

103 lines (81 loc) · 4.59 KB
Raw

made-with-python

Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval

Preprint:

https://www.arxiv.org/abs/2409.09430

Dataset: a subset of MedMNIST V2

Project Image

Feature extractors:

  • VGG19
  • ResNet50
  • DenseNet121
  • EfficientNetV2M
  • MedCLIP
  • BioMedClip
  • OpenCLIP
  • CONCH
  • UNI

Similarity:

Cosine similarity index

Evaluation:

  • mAP@5
  • MMV@5
  • ACC@1
  • ACC@3
  • ACC@5

Results:

  • Main results in the manuscript
  • More detailed results are available in "Supplementary Materials"
  • Selective results based on ACC@1:
ACC@1 for different datasets based on the models and input image size

How to run:

  • For pre-trained CNNs for 2D datasets run main_2D_CNN.py
  • For pre-trained CNNs for 2D datasets (if your memory is limited to load the entire data) run main_2D_CNN_memory_safe
  • For pre-trained foundation models for 2D datasets run main_2D_foundation.py
  • For pre-trained foundation models for 2D datasets (if your memory is limited to load the entire data) run main_2D_foundation_memory_safe
  • For pre-trained models (CNN or foundation) for 3D datasets run main_3D.py
  • To create bar charts run plot.py

Used repositories/sources:

Citation:

@article{mahbod2024evaluating,
  title={Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval},
  author={Mahbod, Amirreza and Saeidi, Nematollah and Hatamikia, Sepideh and Woitek, Ramona},
  journal={arXiv preprint arXiv:2409.09430},
  year={2024}
}

Contact:

Amirreza Mahbod [email protected]

Main References:

  • VGG19: Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
  • ResNet50: He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  • DenseNet121: Huang, Gao, et al. "Densely connected convolutional networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
  • EfficientNetV2M: Tan, Mingxing, and Quoc Le. "Efficientnetv2: Smaller models and faster training." International conference on machine learning. PMLR, 2021.
  • MedCLIP: Wang, Zifeng, et al. "Medclip: Contrastive learning from unpaired medical images and text." arXiv preprint arXiv:2210.10163 (2022).
  • BioMedClip: Zhang, Sheng, et al. "BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs." arXiv preprint arXiv:2303.00915 (2023).
  • OpenCLIP: Cherti, Mehdi, et al. "Reproducible scaling laws for contrastive language-image learning." Proceedings of the IEEE/CVF Conference on - Computer Vision and Pattern Recognition. 2023.
  • CONCH: Lu, Ming Y., et al. "Visual language pretrained multiple instance zero-shot transfer for histopathology images." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.
  • UNI: Chen, Richard J., et al. "Towards a general-purpose foundation model for computational pathology." Nature Medicine 30.3 (2024): 850-862.