This repository is the official implementation of Out-of-Distribution Image Classification task in the Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks.
Version Control of Python libraries in environment.yml file. To create a virtual environment:
conda env create -f environment.yml
Install tlllib
python3 -m pip install -i https://test.pypi.org/simple/ tllib==0.4
We include shell scripts to benchmark OOD generalization performance for image classification: for robustness to style and structure variations on ImageNet variants (ImageNet Sketch, Renditions, Adversarial, and V2), and synthetic-to-real generalization on VisDA2017.
To evaluate on ImageNet variants, run:
bash eval_classifier_ood.sh
To train on VisDA (syn), and evaluate on VisDA (real), run:
bash eval_classifier_ood.sh
Update the following variables as appropriate:
BACKBONE, VALID_LABELS_INA, VALID_LABELS_INR, CHECKPOINT, DATA_DIR