Fallback llms with AsyncCallbackHandler is not consistent with Regular LLM

[bharath-muthineni]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

for chain_output_key, chain_output_val in outputs.items():
                    if isinstance(chain_output_val, list):
                        self.dataset.add_records(
                            records=[
                                {
                                    "fields": all_fields | {
                                        self.supported_fields[0]: prompt,
                                        self.supported_fields[1]: self.thoughts.get(run_id, " \n"),
                                        self.supported_fields[2]: output["text"].strip(),
                                    },
                                    "metadata": context_run,
                                    "external_id": external_id
                                }
                                for prompt, output in zip(
                                    prompts, chain_output_val  # type: ignore
                                )
                            ]
                        )


This is how I update my callback handler.

Description

I'm trying to use the llm with fallback method to handle any unexpected fallbacks in my chain.
One issue that I've been seeing is that when I pass my callback handler, without the fallbacks, the output I receive to my on_chain_end method is a dict with my output_key.
However, when I use the fallback llm, the ouput is an AIMessage.

Is this an issue which was fixed in later version of langchain? My current version of langchain stack is:

1. langchain==0.0.351
2. langchain-community==0.0.6
3. langchain-core==0.1.3
4. langchain-experimental==0.0.42

Please help! 🙏

System Info

langchain==0.0.351
langchain-community==0.0.6
langchain-core==0.1.3
langchain-experimental==0.0.42

[dosubot[bot]]

Hey @phenom-xplus! 👋 I'm here to help you with bugs, questions, and even becoming a contributor. Let's tackle this issue together while we wait for a human maintainer.

The difference in the output format of your on_chain_end method when using fallback LLMs is due to how the _convert_dict_to_message function processes the response. Specifically, it checks for the presence of function_call and tool_calls in the response dictionary and includes them in the additional_kwargs of the AIMessage.

Here is the relevant code snippet:

def _convert_dict_to_message(_dict: Mapping[str, Any]) -> BaseMessage:
    role = _dict["role"]
    if role == "user":
        return HumanMessage(content=_dict["content"])
    elif role == "assistant":
        content = _dict.get("content", "") or ""
        additional_kwargs = {}
        if _dict.get("function_call"):
            additional_kwargs["function_call"] = dict(_dict["function_call"])
        if _dict.get("tool_calls"):
            additional_kwargs["tool_calls"] = _dict["tool_calls"]
        return AIMessage(content=content, additional_kwargs=additional_kwargs)
    elif role == "system":
        return SystemMessage(content=_dict["content"])
    elif role == "function":
        return FunctionMessage(content=_dict["content"], name=_dict["name"])
    else:
        return ChatMessage(content=_dict["content"], role=role)

When the role is "assistant", the function checks for function_call and tool_calls in the dictionary and adds them to additional_kwargs. This results in an AIMessage with fallbacks. If these keys are not present, the resulting message will not include these fallbacks ^[1].

Additionally, the on_chain_end method in the Chain class processes the outputs and invokes the callback manager's on_chain_end method, which might also influence the final output format ^[2].

In the context of the AsyncCallbackHandler, the on_chain_end method updates the response with the outputs and custom callback metadata, which might include additional information such as function_call and tool_calls, leading to the observed differences in the output format ^[3].

To continue talking to Dosu, mention @dosu.