Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TextReading] Text extractions grading recall bug #755

Closed
enoriega opened this issue Jan 16, 2024 · 0 comments
Closed

[TextReading] Text extractions grading recall bug #755

enoriega opened this issue Jan 16, 2024 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@enoriega
Copy link
Member

Recall is being computed incorrectly and sometimes it exceeds 1.0.

Example:

{
  "num_manual_annotations": 241,
  "yield_": 400,
  "correct_extractions": 356,
  "recall": 1.4771784232365146,
  "precision": 0.8899999999999777,
  "f1": 1.1107644305302822
}

Attached are the files used to compute these numbers
expected_cosmos_ground_truth.json
extractions.json
annotations.json

@enoriega enoriega added the bug Something isn't working label Jan 16, 2024
@enoriega enoriega added this to the [DARPA] Milestone 11 milestone Jan 16, 2024
@enoriega enoriega self-assigned this Jan 16, 2024
enoriega added a commit that referenced this issue Jan 16, 2024
github-actions bot added a commit that referenced this issue Jan 16, 2024
## Summary of Changes
Updated the TR grading code to stop double counting automated
extractions which lead to an incorrect amount of true positives.

## Expanded explanation
There is a many to one relationship between SKEMA extractions and manual
annotations. Multiple extractions can match an annotation (be correct).
To compute P, R and F1, all the extractions associated to a manual
annotation should be counted as a single true positive.

This change addresses this issue.

### Related issues

Resolves 755 e017c99
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant