_log_detection_eval_metrics reads precision array from coco_eval wrong

[deuscovrigus]

When computing the per class precision of a JsonDataset instance using accumulated results in a COCOeval instance, the category order gets mixed up if the JsonDataset classes stored in the corresponding json file are not already sorted.

In this line the precision array for each class (coming from the coco_eval instance) is read at index cls_ind -1. Everything would be fantastic if the order of the catIds in COCOeval class would be the same as the order of the json_dataset.classes However, this is not guaranteed, and most often is not the case.
Note that the category ids are sorted in the COCOeval class here.

if not cocoGt is None:
     self.params.imgIds = sorted(cocoGt.getImgIds())
     self.params.catIds = sorted(cocoGt.getCatIds())

The COCOeval.accumulate function proceeds to allocating a Nd array with the third dimension corresponding to the sorted catIds. The order is given by the k_list here.

setK = set(catIds)
setA = set(map(tuple, _pe.areaRng))
setM = set(_pe.maxDets)
setI = set(_pe.imgIds)
# get inds to evaluate
k_list = [n for n, k in enumerate(p.catIds)  if k in setK]

The JsonDataset class does not sort the category ids as seen in its __init__ here.

 category_ids = self.COCO.getCatIds()
 categories = [c['name'] for c in self.COCO.loadCats(category_ids)]
 self.category_to_id_map = dict(zip(categories, category_ids))
 self.classes = ['__background__'] + categories

Therefore the problem occurs in _log_detection_eval_metrics by attempting to loop over the indexes of the unsorted class names and attempt to read the same index minus one (accounting for the bknd class) from the third dimension of the precision array from coco_eval. However, as stated above, coco_eval.eval['precision'] is filled in the order of sorted category ids.

for cls_ind, cls in enumerate(json_dataset.classes):
        if cls == '__background__':
            continue
        # minus 1 because of __background__
        precision = coco_eval.eval['precision'][
            ind_lo:(ind_hi + 1), :, cls_ind - 1, 0, 2]

In order to correct this, a mapping between the JsonDataset class indexes (contiguous_category_id) to sorted class ids can be created in the JsonDataset.__init__ as such

sorted_cats = sorted(self.COCO.getCatIds()) 
self.contiguous_category_id_to_sorted_json_id = {                                                                                                                                                                    
   self.json_category_id_to_contiguous_id[cat_id]: i                                                                                                                                                                
   for i, cat_id in enumerate(sorted_cats)                                                                                                                                                                          
   }

Then, when reading the coco_eval precision array modify this line as such:

precision = coco_eval.eval['precision'][                                                                                                                                                                             
    ind_lo:(ind_hi + 1), :, json_dataset.contiguous_category_id_to_sorted_json_id[cls_ind], 0, 2]

Note that there is no need to subtract the one from cls_ind as the mapping contiguous_category_id_to_sorted_json_id does not include the background class