Evaluating COCO base

[atiszabo]

Hi,

I am trying to evaluate the base checkpoint for COCO to see AP results for each class before few-shot training.

• With --meta_test it needs the saved mean_class_attentions.pkl from training, which I can extract but it seems to be an error with the provided checkpoint. Before saving it, I printed out the keys in the dict for the classes and there were only 3 of them:
  (I've extracted these from scratch deleting all .pt or .pkl files that were generated)

after class filtering, there are 197098 images...

197098 roidb entries
Loading pretrained weights from data/pretrained_model/resnet101_caffe.pth
loading checkpoint save_models/COCO/coco_metarcnn_200_20.pth
loaded checkpoint save_models/COCO/coco_metarcnn_200_20.pth
dict_keys([11, 23, 17])

with proposed new checkpoint for COCO [Dec-15 2020] :

after class filtering, there are 197098 images...

197098 roidb entries
Loading pretrained weights from data/pretrained_model/resnet101_caffe.pth
loading checkpoint save_models/COCO/coco_metarcnn_200_20.pth
loaded checkpoint save_models/COCO/coco_metarcnn_200_20.pth
dict_keys([13])

• Without --meta_test flag, meaning that no mean_class_attentions.pkl is required it also fails:

RuntimeError: size mismatch, m1: [300 x 2048], m2: [4096 x 324] at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generic/THCTensorMathBlas.cu:249

Data seems fine and in place as expected.
@YoungXIAO13 Could you help with this issue? Can you verify that COCO checkpoint is correct? Do you have checkpoints that are already trained with few-shot part?

@hjraad

[atiszabo]

@Skaldak in #19 (comment) you mention that you could get a reasonable base model, could you share that checkpoint with us? Unfortunately, we cannot afford to train on COCO base fully for several days.