Skip to content

Latest commit

 

History

History
77 lines (60 loc) · 3.75 KB

README.md

File metadata and controls

77 lines (60 loc) · 3.75 KB
Raw

Salient ImageNet

This repository provides code for the ICLR 2022 paper titled Salient ImageNet: How to discover spurious features in deep learning?.

Prerequisites

  • Python >= 3.7
  • Pytorch >= 1.9.0
  • PIL >= 8.4.0
  • clip. Can be installed using pip install git+https://github.com/openai/CLIP.git
  • timm >= 0.4.12
  • pandas >= 1.3.1
  • numpy >= 1.21.2
  • cv2 >= 4.5.3
  • robustness

Setup

  • Load the Robust Resnet-50 model using the command given below:
    wget -O models/robust_resnet50.pth https://www.dropbox.com/s/knf4uimlqsi1yz8/imagenet_l2_3_0.pt?dl=0
  • Load the Salient ImageNet dataset using this link.
  • Alternatively, download the Salient ImageNet dataset on a remote server using the command given below:
    curl -L https://umd.app.box.com/shared/static/eyji8leh2lemhbxovgny9ywc9is53ibr -o salient_imagenet_dataset.zip
  • Unzip the Salient Imagenet zip file using: unzip salient_imagenet_dataset.zip
  • Update IMAGENET_PATH in dataset_paths.py to the ImageNet dataset (The last folder in IMAGENET_PATH should be /ILSVRC2012/).
  • NOTE: Some ImageNet versions require the meta.bin file to be placed in the ImageNet folder ending with /ILSVRC2012/. For this purpose, you can use the meta.bin file provided in this repo.

Visualizing class and feature pairs

  • Run visualize_class_feature_pair.ipynb to visualize the feature, class and obtain the Mechanical Turk Worker annotations.
  • Specify class_index, feature_index in the jupyter notebook to visualize features in Section J of the paper.
  • Example for class_index = 325, feature_index = 595 given below: images heatmaps attacks

Visualizing Salient Imagenet

  • Run visualize_salient_imagenet.ipynb to visualize the images and heatmaps in Salient Imagenet dataset.
  • Specify class_index, feature_index in the jupyter notebook to visualize features in Section J of the paper.
  • Example for class_index = 309, feature_index = 1941 given below: images heatmaps

Visualizing class

  • Run visualize_class.ipynb to visualize the class and top-5 most predictive features for the class.
  • Specify class_index in the jupyter notebook.
  • Example for class_index = 325 given below:
    Object name(s): sulphur butterfly, sulfur butterfly
    Object supercategory: inverterbrates
    Object definition: any of numerous yellow or orange butterflies
    Object wikipedia link(s): https://en.wikipedia.org/wiki/Coliadinae
    Object images from ImageNet validation set:
    samples

Testing sensitivity of pretrained models

  • The pretrained models to evaluate are given in the list model_name_list and standard deviation of Gaussian noise in the list noise_stds_list.
  • Run evaluate_sensitivity_single_feature.ipynb to test the sensitivity of pretrained models to all 5000 features.
  • Run evaluate_sensitivity_core_features.ipynb to test the sensitivity of pretrained models to core features and compute spurious accuracy.
  • Run evaluate_sensitivity_spurious_features.ipynb to test the sensitivity of pretrained models to core features and compute core accuracy.

Citations

@inproceedings{
  singla2022salient,
  title={Salient ImageNet: How to discover spurious features in Deep Learning?},
  author={Sahil Singla and Soheil Feizi},
  booktitle={International Conference on Learning Representations},
  year={2022},
  url={https://openreview.net/forum?id=XVPqLyNxSyh}
}