Progressive Token Length Scaling in Transformer Encoders
for Efficient Universal Segmentation
(official implementation)

Abhishek Aich  Yumin Suh  Samuel Schulter  Manmohan Chandraker 

NEC Laboratories, America

Key Features

Efficient encoder design to reduce computational cost of Mask2Former models

• Simple. Easily pluggable to Mask2Former-style models.
• Effective. Flexible in reducing encoder cost by 50-75% with little to no segmentation performance degradation.
• General. Evaluated with various backbones, pre-trained weights as well as DETR model (DINO).

Model Framework

Installation

• Our project is developed on Mask2Former. Please follow their installation instructions as well as dataset instructions. You may also refer to the requirements.txt.

Download pretrained weights

• Create a folder named pretrained_ckpt in the root folder using mkdir pretrained_ckpt.
• Please use the analyze_model code from Mask2Former to compute GFLOPs reported in the paper.
• Download our pretrained weights and put them in pretrained_ckpt.
  Our pretrained weights includes PRO-SCALE (configuration (3,3,3)) based Mask2Former models with corresponding backbone and trained on corresponding dataset:

Model Name	Backbone	Dataset	Config	PQ (%)
r50_proscale_3_3_3_coco	Res50	COCO	r50_proscale_3_3_3_coco	51.44
swint_proscale_3_3_3_coco	SWIN-T	COCO	swint_proscale_3_3_3_coco	52.88

Example run command (please set the dataset and output directory path):

DETECTRON2_DATASETS=/path/to/dataset/folder/ python3 test_net.py \
    --num-gpus 1 \
    --eval-only \
    --config-file configs/res50_proscale_3_3_3_coco.yaml \
    MODEL.WEIGHTS pretrained_ckpt/r50_proscale_3_3_3_coco.pth \
    OUTPUT_DIR /path/to/output/folder

Contact

If you have any questions, don't hesitate to contact me at [email protected].

Acknowledgement

This repository was built on top of Detectron2 and Mask2Former. We thank and acknowledge their effort. Please also see our contemporary work ECO-M2F (paper, code) on efficient transformer encoders.

Citing PRO-SCALE

If you find our work helpful for your research, please consider citing the following BibTeX entry.

@article{aich2024progressive,
  title={Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation},
  author={Aich, Abhishek and Suh, Yumin and Schulter, Samuel and Chandraker, Manmohan},
  journal={arXiv preprint arXiv:2404.14657},
  year={2024}
}