Can't reproduce the results on PC-459 dataset using the provided ckpt.

[SuleBai]

Hi @cornettoyu , thanks for your great work.

While using the provided checkpoint(ConvNeXt-Large fcclip_cocopan.pth) to reproduce the results on OVS task, I can get similar results on all the other datasets except for the PC-459 dataset. The log is here
pc459-log.txt.

	A-847	A-150	PC-459	PC-59	VOC-20	PAS-21
paper	14.8	34.1	18.2	58.4	95.4	81.8
ckpt	14.7	34.0	13.8	58.4	95.3	81.8

I did follow every step in the instrcution and tried to prepare the PC-459 dataset several times, but I can't get the expected results on PC-459. Now I am confused what is going wrong here.

And the command I use is

python train_net.py  --num-gpus 8   --config-file configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_pc459.yaml   --eval-only MODEL.WEIGHTS ./fcclip_cocopan.pth 

[SuleBai]

Update:
It seems it is due to the detectron2 version. (About how it treat nan.)

If I use the command below to install detectron2(version 0.6+cu111), I get the mIoU around 13.

python -m pip install detectron2 -f \
  https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.9/index.html

However if I use the command below to install detectron2(version 0.6), I get the mIoU around 18.

git clone https://github.com/facebookresearch/detectron2.git
python -m pip install -e detectron2

And after checking the log, I think the issue is due to the 'nan' results, which only happens in the PC-459 dataset. The mIoU 13 takes the nan class into account while the mIoU 18 did not take the nan class into account when calculating the metrics.

It seems very strange.