You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When evaluating multiple query relevance (qrel) sets with pytrec_eval, incorrect scores are generated when one or more qrel sets are empty. For example:
When evaluating multiple query relevance (qrel) sets with pytrec_eval, incorrect scores are generated when one or more qrel sets are empty. For example:
qrel = {
'q1': {
'd1': 0,
'd2': 1,
'd3': 0,
},
'q2': {
},
'q3': {
'd2': 1,
'd3': 1,
},
}
evaluator = pytrec_eval.RelevanceEvaluator(qrel, {'map', 'ndcg'})
evaluator.evaluate(some_valid_b)
In this case, the evaluator provides erroneous scores for 'q2' and relations after q2 i.e q3, without raising any warning, which could mislead users.
The text was updated successfully, but these errors were encountered: