Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TextReading] Text extractions grading recall bug #755

Closed
enoriega opened this issue Jan 16, 2024 · 0 comments
Closed

[TextReading] Text extractions grading recall bug #755

enoriega opened this issue Jan 16, 2024 · 0 comments
Assignees
@enoriega
Labels
bug Something isn't working
Milestone
[DARPA] Milestone 11

Comments

@enoriega
Copy link
Member

Recall is being computed incorrectly and sometimes it exceeds 1.0.

Example:

{
  "num_manual_annotations": 241,
  "yield_": 400,
  "correct_extractions": 356,
  "recall": 1.4771784232365146,
  "precision": 0.8899999999999777,
  "f1": 1.1107644305302822
}

Attached are the files used to compute these numbers
expected_cosmos_ground_truth.json
extractions.json
annotations.json
@enoriega enoriega added the bug Something isn't working label Jan 16, 2024
@enoriega enoriega added this to the [DARPA] Milestone 11 milestone Jan 16, 2024
@enoriega enoriega self-assigned this Jan 16, 2024
enoriega added a commit that referenced this issue Jan 16, 2024
@enoriega
Fixed #755
fa580a0
github-actions bot added a commit that referenced this issue Jan 16, 2024
@github-actions
Fixed #755 (#756)
b880d18 
## Summary of Changes
Updated the TR grading code to stop double counting automated
extractions which lead to an incorrect amount of true positives.

## Expanded explanation
There is a many to one relationship between SKEMA extractions and manual
annotations. Multiple extractions can match an annotation (be correct).
To compute P, R and F1, all the extractions associated to a manual
annotation should be counted as a single true positive.

This change addresses this issue.

### Related issues

Resolves 755 e017c99
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@enoriega enoriega

Labels
bug Something isn't working
Projects
None yet
Milestone
[DARPA] Milestone 11
Development

No branches or pull requests
1 participant
@enoriega