Evaluation Study on SAM 2 for Class-agnostic Instance-level Segmentation

Code repository for the paper titled "Evaluation Study on SAM 2 for Class-agnostic Instance-level Segmentation"

[Paper]

Get Started

Install

1. Download the SAM and SAM2 code from the website. [SAM] and [SAM2]
2. Please follow the [SAM] and [SAM2] to install the enveriment.
3. copy the code in SAM/notebooks into segment-anything/notebooks. Copy the code in SAM2/notebooks into segment-anythings/notebooks

Datasets

SIS：

• ILSO1K: Google Drive
• SOC: Google Drive
• SIS10KBaidu Disk Verification code: hust
• SIP: Google Drive

CIS：

• COD10K: Baidu (password:hust) / Google / Quark; Json files: Baidu (password:hust) / Google
• NC4K: Baidu (password:hust) / Google; Json files: Baidu (password:hust) / Google

SID：

• SOBA dataset can be downloaded from https://github.com/stevewongv/SSIS

DIS：

• DIS5K dataset can be downloaded from https://github.com/xuebinqin/DIS

Evaluation

SIS/CIS/SID:

1. Download the weights of SAM and SAM2 from the website.
2. Put weights into the checkpoints/
3. Use .sh files in notebooks to evaluate:

-->use SAM， auto model， SIS task as example：\ modify the dataset root in SAM/notebooks/run_auto_saliency.sh

# SIS task / SAM / auto model
$ cd notebooks/
$ bash run_auto_saliency.sh

Note that, [sam/sam2]_[your mode]shadow.sh means the SID task. For CIS task, you can change the dataset path and save_json path in [sam/sam2][your mode]cos10k.py or [sam/sam2][your mode]_nc4k.py, then

python  [sam/sam2]_[your mode]_cos10k.py
# or 
python  [sam/sam2]_[your mode]_nc4k.py

SIS task use he AP70 metric instead of AP75.

After running the [sam/sam2]_[your mode]_saliency.sh, please change the path in run_AP70.sh, then

$ bash run_AP70.sh

DIS:

1. To Get the predicted results of SAM.

• automatic prompts mode:

following the [SAM], masks can be generated for images from the command line:

python scripts/amg.py --checkpoint <path/to/checkpoint> --model-type <model_type> --input <image_or_folder> --output <path/to/output>

then, selecting the most suitable foreground mask, use a maximum Intersection over Union (IoU)

cd DIS/script
python3 findMaxIoUMask.py 

• bounding box prompt mode:

cd DIS/SAM
python3 test_with_box_prompt_floder.py 

2. To Evaluate the predicted results.

cd DIS/metrics
python3 test_metrics.py 
python3 hce_metric_main.py

Qualitative Results of SIS, CIS and SID

1. SAM2 sam2_demo
2. SAM sam_demo

Citation

If you find our work useful for your research or applications, please cite using this BibTeX:

@article{zhang2024evalsam2,
      title={Evaluation Study on SAM 2 for Class-agnostic Instance-level Segmentation}, 
      author={Zhang, Tiantian and Zhou, Zhangjun and Pei, Jialun},
      journal={arXiv preprint arXiv:2409.02567},
      year={2024},
      url={https://arxiv.org/abs/2409.02567} 
}

Acknowledgement

Thanks for the efforts of the authors involved in the Segment Anything, Segment Anything 2 and UnderwaterSAM2Eval.