Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: return streaming errors as an event formatted for openai's client #2668

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

drbh
Copy link
Collaborator

@drbh drbh commented Oct 18, 2024

This PR adds the functionality to emit errors as formatted events.

Note the changes look large because of a new indent however the main logic that changed is the addition of

Err(err) => {
    let error_event: ErrorEvent = err.into();
    let event = Event::default().json_data(error_event).unwrap_or_else(|e| {
        InferError::StreamSerializationError(e.to_string()).into()
    });
    yield Ok::<Event, Infallible>(event);
    break;
}

this change enables the openai client to pull the error message from the stream and correctly surface it to the user

example script (throws with change included)

import openai
from openai import OpenAI

client = OpenAI(
    api_key="key",
)

client.base_url = "http://127.0.0.1:3000/v1"

try:
    completion = client.chat.completions.create(
        messages=[
            {
                "role": "user",
                "content": "test " * 32_000,
            }
        ],
        model="tgi",
        stream=True,
    )

    for index, message in enumerate(completion):
        print(f"Message {index}: {message}")
except openai.APIError as e:
    print(e)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant