Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating make-me-say to be compatible with Solvers #1546

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

lennart-finke
Copy link

@lennart-finke lennart-finke commented Aug 18, 2024

This PR refactors make-me-say to be compatible with the Solvers API.

Instead of passing three completion functions, the eval is now passed one solver as the con artist and two completion functions as the mark and summary model respectively. We still assume gpt-4-32k and gpt-3.5-turbo-16k as defaults, as #1530 is not yet merged. (Edit: Reviewer suggested adding gpt-4o-mini as default instead and changing the registry ourselves.)


Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

  • I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

.gitignore Show resolved Hide resolved
@lennart-finke
Copy link
Author

Thanks for the comments @danesherbs! Addressed them and reran tests, ready for further review or merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants