Efficient retrieval of unevaluated span documents in Phoenix

[firetix]

Discussed in #4346

^{Originally posted by firetix August 23, 2024}
I'm encountering an issue while attempting to retrieve unevaluated Documents from Phoenix. The documented methods evals['EVALUATOR'].score is None and evals['EVALUATOR'].label is None are not yielding the expected results. Additionally, DOCUMENT_SCORE = "document.score" doesn't seem to be effective for retrieving scores.
My objective is to efficiently retrieve the following metrics:

NDCG (Normalized Discounted Cumulative Gain)
Hit rate
Overall score

Here's the current implementation:

import phoenix as px

project_name = "rag"
evaluation_name = "relevance_evaluator"
phoenix_endpoint = "http://localhost:7442"

query = (
    SpanQuery()
    .select("trace_id", **INPUT)
    .explode("retrieval.documents")
)

spans_df = px.Client(endpoint=phoenix_endpoint).query_spans(query, project_name=project_name)

I've attempted to use the SpanQuery to retrieve the necessary data, but I'm unable to filter for unevaluated documents effectively. Has anyone encountered a similar issue or can suggest an optimal approach to achieve this?
Any insights on best practices for querying unevaluated documents and retrieving specific metrics in Phoenix would be greatly appreciated. Thank you in advance for your assistance.

[mikeldking]

Hey @firetix - good question. In general I find it easier to just setup a job that runs on a cadence and to query by time rather than "not evaluated" spans as that can be a pretty expensive query as you get more and more spans. We have an example of this here: https://github.com/Arize-ai/phoenix/tree/main/examples/cron-evals

I know some people have also used a OTEL span processor to evaluate spans as they get produced but I don't have an example of that.