Skip to content
This repository has been archived by the owner on Oct 2, 2024. It is now read-only.

Create base annotator test for annotator runner #466

Merged
merged 9 commits into from
Jul 12, 2024
Merged

Conversation

bkorycki
Copy link
Contributor

This creates a new base class InteractionAnnotatorTest. It's main distinctions from PromptResponseTest:

  • A test must specify an AnnotationType. Unlike regular tests which interface with a list of text completions SUTResponse, tests for annotators expect a certain output from an Annotator. We can build a mechanism later to check for test-Annotator compatibility in the runner.
  • It produces test items that model a single-turn SUT interaction (i.e. a prompt + SUT completion pair). This is better aligned with the input structure of annotator objects. Plus, I believe it makes more sense to let Annotators decide how to format their own request, instead of combining a prompt + completion into a single TestItem in the test itself.

@bkorycki bkorycki requested a review from tsunamit July 10, 2024 16:27
@bkorycki bkorycki requested a review from a team as a code owner July 10, 2024 16:27
Copy link

github-actions bot commented Jul 10, 2024

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@bkorycki bkorycki merged commit 25da305 into rvt/lgeval Jul 12, 2024
4 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Jul 12, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants