ChatOpenAI + HF TGI + Llama3.1 tool calling not working #26342
Replies: 2 comments 4 replies
-
|
Hello, @avfranco-br! I'm here to help you with any bugs, questions, or contributions you might have. Let's tackle this together! The error "TypeError: string indices must be integers, not 'str'" occurs because the code is attempting to access a string using a string index, which is not allowed in Python. This issue arises in the The problem likely lies in the structure of the response dictionary from the Meta-Llama-3.1-70B-Instruct model, which might differ from the expected structure. Specifically, the code expects certain keys and values in the response dictionary, and if the structure is different, it can lead to this type of error. Here is the relevant part of the code that could be causing the issue: def _convert_dict_to_message(_dict: Mapping[str, Any]) -> BaseMessage:
"""Convert a dictionary to a LangChain message.
Args:
_dict: The dictionary.
Returns:
The LangChain message.
"""
role = _dict.get("role")
name = _dict.get("name")
id_ = _dict.get("id")
if role == "user":
return HumanMessage(content=_dict.get("content", ""), id=id_, name=name)
elif role == "assistant":
content = _dict.get("content", "") or ""
additional_kwargs: Dict = {}
if function_call := _dict.get("function_call"):
additional_kwargs["function_call"] = dict(function_call)
tool_calls = []
invalid_tool_calls = []
if raw_tool_calls := _dict.get("tool_calls"):
additional_kwargs["tool_calls"] = raw_tool_calls
for raw_tool_call in raw_tool_calls:
try:
tool_calls.append(parse_tool_call(raw_tool_call, return_id=True))
except Exception as e:
invalid_tool_calls.append(
make_invalid_tool_call(raw_tool_call, str(e))
)
return AIMessage(
content=content,
additional_kwargs=additional_kwargs,
name=name,
id=id_,
tool_calls=tool_calls,
invalid_tool_calls=invalid_tool_calls,
)
elif role == "system":
return SystemMessage(content=_dict.get("content", ""), name=name, id=id_)
elif role == "function":
return FunctionMessage(
content=_dict.get("content", ""), name=cast(str, _dict.get("name")), id=id_
)
elif role == "tool":
additional_kwargs = {}
if "name" in _dict:
additional_kwargs["name"] = _dict["name"]
return ToolMessage(
content=_dict.get("content", ""),
tool_call_id=cast(str, _dict.get("tool_call_id")),
additional_kwargs=additional_kwargs,
name=name,
id=id_,
)
else:
return ChatMessage(content=_dict.get("content", ""), role=role, id=id_) # type: ignore[arg-type]To fix this issue, you need to ensure that the response dictionary from the Meta-Llama-3.1-70B-Instruct model matches the expected structure. You might need to add additional checks or modify the response parsing logic to handle the specific structure of the Meta-Llama-3.1-70B-Instruct model's response [1]. The expected structure of the response dictionary from the Meta-Llama-3.1-70B-Instruct model hosted on HF via TGI is as follows: {
"id": "chatcmpl_id",
"object": "chat.completion",
"created": 1721875529,
"model": "meta-llama-3.1-70b-instruct-072424",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "To calculate the result of 36939 multiplied by 8922.4, I get:\n\n36939 x 8922.4 = 329,511,111.6",
},
"finish_reason": "stop",
"logprobs": None,
}
],
"usage": {
"prompt_tokens": 30,
"completion_tokens": 36,
"total_tokens": 66
}
}Ensure that the response from the Meta-Llama-3.1-70B-Instruct model adheres to this structure [2]. |
Beta Was this translation helpful? Give feedback.
-
|
@dosu, I'm not quite sure that above is the problem. Reason why is looking at the LangSmith trace the error seems to happen before or trying to call the tool. |
Beta Was this translation helpful? Give feedback.
-
|
@dosu, here is an entire copy of the message error: |
Beta Was this translation helpful? Give feedback.
-
|
The error "TypeError: string indices must be integers, not 'str'" occurs because the code is attempting to access a string using a string index, which is not allowed in Python. This issue arises in the The problem likely lies in the structure of the response dictionary from the Meta-Llama-3.1-70B-Instruct model, which might differ from the expected structure. Specifically, the code expects certain keys and values in the response dictionary, and if the structure is different, it can lead to this type of error. Here's the relevant part of the code that could be causing the issue: def _convert_delta_to_message_chunk(delta: Mapping[str, Any]) -> MessageChunk:
"""Convert a delta to a message chunk."""
tool_call_chunks = [
ToolCallChunk(
name=rtc["function"].get("name"),
arguments=rtc["function"].get("arguments"),
)
for rtc in delta.get("tool_calls", [])
]
return MessageChunk(
content=delta.get("content", ""),
tool_calls=tool_call_chunks,
)In this code, To fix this issue, you need to ensure that the response dictionary from the Meta-Llama-3.1-70B-Instruct model matches the expected structure. You might need to add additional checks or modify the response parsing logic to handle the specific structure of the Meta-Llama-3.1-70B-Instruct model's response. Ensure that the response from the Meta-Llama-3.1-70B-Instruct model adheres to the expected structure, particularly the For example, you can add a check to ensure def _convert_delta_to_message_chunk(delta: Mapping[str, Any]) -> MessageChunk:
"""Convert a delta to a message chunk."""
tool_call_chunks = [
ToolCallChunk(
name=rtc["function"].get("name") if isinstance(rtc.get("function"), dict) else None,
arguments=rtc["function"].get("arguments") if isinstance(rtc.get("function"), dict) else None,
)
for rtc in delta.get("tool_calls", [])
]
return MessageChunk(
content=delta.get("content", ""),
tool_calls=tool_call_chunks,
)This adjustment ensures that the code only attempts to access |
Beta Was this translation helpful? Give feedback.
-
|
We encountered the same Issue. We are hosting as well the Llama 3.1 70B with HF TGI and we found that the Tool Calling is not working. I tried to actually build the prompt myself and use the LLM directly instead of the ChatModel Version of HF. Here the code for that: from transformers import AutoTokenizer
from langchain_core.messages import BaseMessage, ChatMessage, HumanMessage, AIMessage, SystemMessage, ToolMessage
from typing import Dict, List
def _convert_message_to_chat_message(message: BaseMessage) -> Dict:
if isinstance(message, ChatMessage):
return dict(role=message.role, content=message.content)
elif isinstance(message, HumanMessage):
return dict(role="user", content=message.content)
elif isinstance(message, AIMessage):
tool_calls = None
if "tool_calls" in message.additional_kwargs:
tool_calls = [
{
"function": {
"name": tc["function"]["name"],
"arguments": tc["function"]["arguments"],
}
}
for tc in message.additional_kwargs["tool_calls"]
]
# Only include tool_calls if they are present
result = {
"role": "assistant",
"content": message.content,
}
if tool_calls:
result["tool_calls"] = tool_calls
return result
elif isinstance(message, SystemMessage):
return dict(role="system", content=message.content)
elif isinstance(message, ToolMessage):
return {
"role": "tool",
"content": message.content,
"name": message.name,
}
else:
raise ValueError(f"Got unknown type {message}")
# Function to convert list of Langchain messages to TGI's chat message format
def convert_messages_and_apply_chat_template(messages: List[BaseMessage], tools: List[Dict], model_id: str):
# Step 1: Convert Langchain messages to dictionaries
converted_messages = [_convert_message_to_chat_message(message) for message in messages]
# Ensure tools is not None
if tools is None:
tools = []
# Step 2: Initialize the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Step 3: Apply the chat template with the converted messages and tools
try:
inputs = tokenizer.apply_chat_template(
converted_messages,
tools=tools, # Ensure tools are a valid list
tokenize=False,
add_generation_prompt=True,
return_dict=False,
)
except Exception as e:
print(f"Error occurred: {e}")
return None
return inputs
# Example usage:
langchain_messages = [HumanMessage(content="How is the current weather in Hamburg?")]
# Ensure tools is not None
tools = [
{
"name": "get_current_weather",
"description": "Get the current weather",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"},
"format": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"}
},
"required": ["location", "format"]
}
}
]
# Call the function
tokenized_input = convert_messages_and_apply_chat_template(langchain_messages, tools, model_id)
print(tokenized_input)This is a little bit hacky but works for now. I guess one would need to write a CustomChatModel to convert the input before, or the TGI gets adapted to parse the output the right way. Happy to hear if someone found a better solution. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @kteppris for replying and sharing your code! I will try it. I’ve also experimented with the HF ChatHuggingFace class that the tool calling worked but it entered in an infinite loop. Here’s the code. |
Beta Was this translation helpful? Give feedback.
-
Checked other resources
Commit to Help
Example Code
Description
I am trying to run the above code using ChatOpenAI class with Meta-Llama-3.1-70B-Instruct hosted on HF via TGI. Here's the LangSmith run https://smith.langchain.com/public/916f2317-cd69-44f2-955a-a0325ebd2e0c/r
The error happen when calling this code:
result = self.runnable.invoke(state) # Invoke the LLMError: ```
File "/Users/avfranco/.pyenv/versions/llama/lib/python3.11/site-packages/langchain_openai/chat_models/base.py", line 254, in
name=rtc["function"].get("name"),
~~~^^^^^^^^^^^^
TypeError: string indices must be integers, not 'str'
Beta Was this translation helpful? Give feedback.
All reactions