Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation

Muzhi Zhu^1*,   Yang Liu^1*,   Zekai Luo^1*,   Chenchen Jing¹,   Hao Chen¹,   Guangkai Xu¹,   Xinlong Wang²,   Chunhua Shen¹

¹Zhejiang University,   ²Beijing Academy of Artificial Intelligence

NeurIPS 2024

🚀 Overview

📖 Description

We systematically study four crucial elements of applying the Diffusion Model to Few-shot Semantic Segmentation. For each of these aspects, we propose several reasonable solutions and validate them through comprehensive experiments.

Building upon our observations, we establish the DiffewS framework, which maximally retains the generative framework and effectively utilizes the pre-training prior. Notably, we introduce the first diffusion-based model dedicated to Few-shot Semantic Segmentation, setting the groundwork for a diffusion-based generalist segmentation model.

Paper

🚩 Plan

• Release the weights.
• Release the inference code.
• Release the training code.

👻 Getting Started

Installation

Preparing the environment following GenPercept.

Dataset

Preparing the dataset following Matcher

Training

Evaluation

Download the pre-trained model weights from here.

🎫 License

For academic use, this project is licensed under the 2-clause BSD License. For commercial use, please contact Chunhua Shen.

🖊️ Citation

If you find this project useful in your research, please consider to cite:

@article{zhu2024unleashing,
  title={Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation},
  author={Zhu, Muzhi and Liu, Yang and Luo, Zekai and Jing, Chenchen and Chen, Hao and Xu, Guangkai and Wang, Xinlong and Shen, Chunhua},
  journal={arXiv preprint arXiv:2410.02369},
  year={2024}
}

Acknowledgement

SegGPT, Matcher, Marigold, GenPercept