Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add promptflow-evals quality metrics as an alternative to ragas #707

Open
prvenk opened this issue Sep 13, 2024 · 1 comment · May be fixed by #731
Open

Add promptflow-evals quality metrics as an alternative to ragas #707

prvenk opened this issue Sep 13, 2024 · 1 comment · May be fixed by #731
Assignees
Labels
enhancement New feature or request

Comments

@prvenk
Copy link
Collaborator

prvenk commented Sep 13, 2024

We currently leverage some llm based evaluation metrics from ragas: https://github.com/explodinggradients/ragas
namely, llm_context_precision, llm_context_recall and llm_answer_relevance in this function compute_llm_based_score. These are the RAG triad of metrics.

For rag usecases, however we have an alternative llm-as-a-judge framework provided by promptflow-evals (supported by Microsoft and part of promptflow): https://pypi.org/project/promptflow-evals/

This evaluation framework has quality metrics such as relevance that can be leveraged for answer relevance or context precision. It has a targeted prompt for groundedness. promptflow-evals also has other quality metrics such as coherence, style, fluency, similarity. Moreover, the package also can enable inclusion of safety metrics such hate unfairness, violence, sexual among others.

Ideally, this can serve as a replacement for ragas metrics, but we can integrate promptflow-evals first and make a decision about removing ragas in a subsequent issue given many might be using ragas metrics.

@prvenk prvenk self-assigned this Sep 13, 2024
@prvenk prvenk added the enhancement New feature or request label Sep 13, 2024
@prvenk
Copy link
Collaborator Author

prvenk commented Sep 13, 2024

@guybartal @ritesh-modi

@prvenk prvenk linked a pull request Sep 17, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant