🗺 prompt playground

[mikeldking]

As a user of Phoenix I don't want to have to go back to my IDE to iterate on a prompt. I want to be able to use the data stored in Phoenix (spans, dataets) and run them through a prompt.

Use-cases

• Replay a template change on an LLM Span
• Run a template change on a dataset
• Construct an evaluation template on a single chosen production span or Dataset - Workflow is testing your Evals and be able to save as experiment
• Synthetic data Generation - Use to generate synthetic data, add columns to current rows of data in a dataset, to help create test data

Milestone 1 - LLM Replay

The most important thing to accomplish is the ability to take a specific LLM step (e.g. a span) and to be able to re-play that execution. In the process of re-executing that step, we should record the new response so that we can compare and evaluate.

Out of Scope

• focus on Chat Completions (things with roles) - not completions

Planning

• [playground] PRD #4658
• [playground][spike] API design for chat completions #4600

UI

• [playground][ui] UI skeleton and feature flag #4606
• [playground][ui] credential storage #4885
• [playground][ui] edit message role #4857
• [playground] llm span to playground parameter parsing utils #4886
• [playground][ui] edit model (e.g. model selector) #4887
• [playground][ui] mustache template editor for codemirror #4888
• [playground][ui] f-string vs mustache template formatting toggle #4889
• [playground][ui] f-string variable parsing utility function #4890
• [playground][ui] mustache template variable parsing utility function
• [playground][ui] template variables (e.g. inputs) UI and storage #4891
• [playground][ui] message re-ordering via drag and drop #4892
• [playground][ui] add new tool UI #4902
• [playground][ui] UI for metrics for a chat completion (token count, latency)
• [playground][ui] dataset input mode #4803
• [playground][ui] display playground instances in "alphabetic" order #4949
• [playground][ui] switch Input to be singular across all prompts / instances #4952
• [playground][ui] Provide a back button if coming from a span #4953

API

• [playground][api] prototype a subscription for chat completion #4773
• [playground][api] use mutations to submit a chat completion #4774
• [playground][gql] invocation parameters to chat completion #4858
• [playground][api] Models and vendors listing resolver #4859
• [playground][api] websocket authentication #4793
• [playground][gql] LLM span recording of chat completions (persistence) #4856
• [playground][gql] server side messages input and roles #4893
• [playground][backend] span recording in the DB in the playground project #4894
• [playground][gql] model input to chat completion #4900
• [playground][gql] tool definition input to chat completion
• [playground][gql] support for output schema (e.g. JSON mode) #4901
• [playground][gql] support for Azure OpenAI #4895
• [playground][gql] support for Anthropic #4897
• [playground][gql] support for Vertex
• [playgound][gql] support for Bedrock #4898
• [playground][gql] surface up metrics for a chat completion (token count, latency)
• [playground][api] dataset example chat completion subscription #4804

Server

• [playground] use wss when https is used to initiate websockets connection #4930

Instrumentation

• [playground][instrumentation] semantic convention for output schema definition (JSON mode) #4896
• [playground][instrumentation] openai support for output schema #4904
• [playground][instrumentation] tool call id semantic conventions #4908
• [playground][instrumentation] capture tool ids in openai python instrumentaiton
• [playground][instrumentation] openai node support for output schema
• [playground][instrumentation] support for model vendor (e.g. system) in the instrumentation

Milestone 2 - Datasets on Playgrounds

Add support to run a set of dataset examples through a prompt

Milestone 3 - Add support for Annotations

As a user, I want the ability to quickly annotate the result of an invocation

[heralight]

Hi!

Enhancement proposal

This feature should be similar to #2462 but with more depth. It would involve a simple button to replicate a query into an edit mode, allowing you to replay it. Additionally, it should offer the possibility to add notes on the result iterations, such as rating the quality, format output, etc., on a scale of 1 to 10.

Goal

The goal is to facilitate quick testing of prompts and inputs, enabling evaluation and visualization of progression.

Thank you,

Alexandre

[mikeldking]

Hi!

Enhancement proposal

This feature should be similar to #2462 but with more depth. It would involve a simple button to replicate a query into an edit mode, allowing you to replay it. Additionally, it should offer the possibility to add notes on the result iterations, such as rating the quality, format output, etc., on a scale of 1 to 10.

Goal

The goal is to facilitate quick testing of prompts and inputs, enabling evaluation and visualization of progression.

Thank you,

Alexandre

@heralight Hey! Thanks for the feedback! We have a ton of features coming out with regards to prompt iteration, notably prompt experiments. Stay tuned. It has evaluations built in

Noted on the replay and the annotations:) will give it some thought. We have a few ideas around replaying data at prompts, but haven't thought about human annotations on different prom versions a ton. Would love to hear more.

[heralight]

Very nice!
my ideal workflow, would be:

• trace some openai calls made from code
• transform some into a replayable prompt, where each modification can be tested and versioned and annoted and ranked.
• prompt can parametized and call from code

best,