ChatOllama not able to product Query output, although ChatGroq can on the same model.

[raghvender-1205]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# Create the query constructor chain
        query_constructor_chain = load_query_constructor_runnable(
            self.openllm.llm,
            # self.openai_llm.llm,
            # enable_limit=True,
            document_contents=self.document_content_description,
            attribute_info=self.metadata_field_info,
            fix_invalid=True,
            allowed_comparators=["eq", "like"]
        )

        retriever = CustomRetriever(
            vectorstore=vs,
            query_constructor=query_constructor_chain,
            translator=QdrantTranslator(metadata_key="metadata") 
        )

        # Use the new combined prompt template
        question_prompt = AppPrompts.QUESTION_WITH_HISTORY_PROMPT
        question_chain = question_prompt | self.openllm.chain()

        if is_rag:
            retriever_chain = RunnablePassthrough.assign(
                context=retriever | (lambda docs: "\n\n".join([d.page_content for d in docs]))
            )
            logger.info(retriever_chain)
            chain = retriever_chain | self.prompt | self.openllm.chain()
        else:
            chain = question_chain | self.prompt | self.openllm.chain()

Description

I am trying to build an RAG using CustomRetriever. I am using Qdrant vectordb and llama3.1-70b as the LLM. This is my Retriever.

class CustomRetriever(BaseRetriever, BaseModel):
    vectorstore: Any = Field(description="The vector store to use for retrieval")
    query_constructor: Any = Field(description="The query constructor to use for generating structured queries")
    translator: QdrantTranslator = Field(default_factory=lambda: QdrantTranslator(metadata_key="payload"))
    
    def get_relevant_documents(self, query: str) -> List[Document]:
        structured_query = self.query_constructor.invoke({"query": query})
        logger.info(structured_query)
        qdrant_filter = self.translator.visit_structured_query(structured_query)

        return self.vectorstore.search(
            structured_query.query,
            search_type="mmr",
            score_threshold=0.7,
            k=20,
            **qdrant_filter[1]
        )

My RAGChain is like this

# Create the query constructor chain
        query_constructor_chain = load_query_constructor_runnable(
            self.openllm.llm,
            # self.openai_llm.llm,
            # enable_limit=True,
            document_contents=self.document_content_description,
            attribute_info=self.metadata_field_info,
            fix_invalid=True,
            allowed_comparators=["eq", "like"]
        )

        retriever = CustomRetriever(
            vectorstore=vs,
            query_constructor=query_constructor_chain,
            translator=QdrantTranslator(metadata_key="metadata") 
        )

        # Use the new combined prompt template
        question_prompt = AppPrompts.QUESTION_WITH_HISTORY_PROMPT
        question_chain = question_prompt | self.openllm.chain()

        if is_rag:
            retriever_chain = RunnablePassthrough.assign(
                context=retriever | (lambda docs: "\n\n".join([d.page_content for d in docs]))
            )
            logger.info(retriever_chain)
            chain = retriever_chain | self.prompt | self.openllm.chain()
        else:
            chain = question_chain | self.prompt | self.openllm.chain()

Now, when I am using ChatGroq

self.llm = ChatGroq(
    model="llama-3.1-70b-versatile",
    temperature=0,
    cache=False
)

This is the query it formed

{
  "output": {
    "query": "usages of the model",
    "filter": {
      "comparator": "eq",
      "attribute": "filename",
      "value": "yolov10"
    }
  }
}

But in ChatOllama it is giving me just a random response in string.

This is my trace.

https://smith.langchain.com/o/c7241417-303a-5408-9d13-fff8d0c5031f/projects/p/b4ef9d3c-f360-4494-b712-b121ded4470e?timeModel=%7B%22duration%22%3A%227d%22%7D&peek=f253fbcd-5af0-4115-8de8-8b911fa59d90

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 23.6.0: Mon Jul 29 21:13:00 PDT 2024; root:xnu-10063.141.2~1/RELEASE_X86_64
Python Version: 3.10.14 (main, May 6 2024, 14:47:20) [Clang 14.0.6 ]

Package Information

langchain_core: 0.2.39
langchain: 0.2.16
langchain_community: 0.2.16
langsmith: 0.1.117
langchain_groq: 0.1.9
langchain_ollama: 0.1.3
langchain_openai: 0.1.23
langchain_text_splitters: 0.2.4

Optional packages not installed

langgraph
langserve

Other Dependencies

aiohttp: 3.10.5
async-timeout: 4.0.3
dataclasses-json: 0.6.7
groq: 0.11.0
httpx: 0.27.2
jsonpatch: 1.33
numpy: 1.26.4
ollama: 0.3.3
openai: 1.44.1
orjson: 3.10.7
packaging: 24.1
pydantic: 2.7.4
PyYAML: 6.0.2
requests: 2.31.0
SQLAlchemy: 2.0.34
tenacity: 8.5.0
tiktoken: 0.7.0
typing-extensions: 4.11.0

[dosubot[bot]]

Hello, @raghvender-1205! How can I assist you today?

ChatOllama is not able to produce the correct query output while ChatGroq can on the same model because Ollama does not support 'tool_choice' yet. This is evidenced by the @pytest.mark.xfail annotations in the test cases, which indicate that the tests are expected to fail due to this lack of support.

@pytest.mark.xfail(
    reason=(
        "Fails with 'AssertionError'. Ollama does not support 'tool_choice' yet."
    )
)
def test_structured_output(self, model: BaseChatModel) -> None:
    super().test_structured_output(model)

@pytest.mark.xfail(
    reason=(
        "Fails with 'AssertionError'. Ollama does not support 'tool_choice' yet."
    )
)
def test_structured_output_pydantic_2_v1(self, model: BaseChatModel) -> None:
    super().test_structured_output_pydantic_2_v1(model)

In contrast, ChatGroq has implemented support for 'tool_choice', allowing it to produce the correct query output. This is demonstrated in the libs/partners/groq/tests/integration_tests/test_standard.py file, where the tool_choice_value is set and used in tests, even though some tests are marked with @pytest.mark.xfail for other reasons.

class TestGroqLlama(BaseTestGroq):
    @property
    def chat_model_params(self) -> dict:
        return {
            "model": "llama-3.1-8b-instant",
            "temperature": 0,
            "rate_limiter": rate_limiter,
        }

    @property
    def tool_choice_value(self) -> Optional[str]:
        """Value to use for tool choice when used in tests."""
        return "any"

Additionally, the ChatGroq class in the libs/partners/groq/langchain_groq/chat_models.py file provides extensive support for various parameters, including tool_choice, which allows it to handle different tools and functions effectively. This is a key reason why ChatGroq succeeds where ChatOllama fails.

def bind_tools(
        self,
        tools: Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]],
        *,
        tool_choice: Optional[
            Union[dict, str, Literal["auto", "any", "none"], bool]
        ] = None,
        **kwargs: Any,
    ) -> Runnable[LanguageModelInput, BaseMessage]:
        """Bind tool-like objects to this chat model.

        Args:
            tools: A list of tool definitions to bind to this chat model.
                Supports any tool definition handled by
                :meth:`langchain_core.utils.function_calling.convert_to_openai_tool`.
            tool_choice: Which tool to require the model to call.
                Must be the name of the single provided function,
                "auto" to automatically determine which function to call
                with the option to not call any function, "any" to enforce that some
                function is called, or a dict of the form:
                {"type": "function", "function": {"name": <<tool_name>>}}.
            **kwargs: Any additional parameters to pass to the
                :class:`~langchain.runnable.Runnable` constructor.
        """
formatted_tools = [convert_to_openai_tool(tool) for tool in tools]
        if tool_choice is not None and tool_choice:
            if tool_choice == "any":
                if len(tools) > 1:
                    raise ValueError(
                        f"Groq does not currently support {tool_choice=}. Should "
                        f"be one of 'auto', 'none', or the name of the tool to call."
                    )
                else:
                    tool_choice = convert_to_openai_tool(tools[0])["function"]["name"]
            if isinstance(tool_choice, str) and (
                tool_choice not in ("auto", "any", "none")
            ):
                tool_choice = {"type": "function", "function": {"name": tool_choice}}
            # TODO: Remove this update once 'any' is supported.
            if isinstance(tool_choice, dict) and (len(formatted_tools) != 1):
                raise ValueError(
                    "When specifying `tool_choice`, you must provide exactly one "
                    f"tool. Received {len(formatted_tools)} tools."
                )
            if isinstance(tool_choice, dict) and (
                formatted_tools[0]["function"]["name"]
                != tool_choice["function"]["name"]
            ):
                raise ValueError(
                    f"Tool choice {tool_choice} was specified, but the only "
                    f"provided tool was {formatted_tools[0]['function']['name']}."
                )
            if isinstance(tool_choice, bool):
                if len(tools) > 1:
                    raise ValueError(
                        "tool_choice can only be True when there is one tool. Received "
                        f"{len(tools)} tools."
                    )
                tool_name = formatted_tools[0]["function"]["name"]
                tool_choice = {
                    "type": "function",
                    "function": {"name": tool_name},
                }

Moreover, the OllamaFunctions class in the libs/experimental/langchain_experimental/llms/ollama_functions.py file shows that while Ollama can bind tools and handle structured outputs, it does not yet support the tool_choice parameter, which is crucial for selecting specific tools during query processing.

class OllamaFunctions(ChatOllama):
    """Function chat model that uses Ollama API."""

    tool_system_prompt_template: str = DEFAULT_SYSTEM_TEMPLATE

    def __init__(self, **kwargs: Any) -> None:
        super().__init__(**kwargs)

    def bind_tools(
        self,
        tools: Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]],
        **kwargs: Any,
    ) -> Runnable[LanguageModelInput, BaseMessage]:
        return self.bind(functions=tools, **kwargs)

    def with_structured_output(
        self,
        schema: Union[Dict, Type[BaseModel]],
        *,
        include_raw: bool = False,
        **kwargs: Any,
    ) -> Runnable[LanguageModelInput, Union[Dict, BaseModel]]:
        """Model wrapper that returns outputs formatted to match the given schema.

        Args:
            schema: The output schema as a dict or a Pydantic class. If a Pydantic class
                then the model output will be an object of that class. If a dict then
                the model output will be a dict. With a Pydantic class the returned
                attributes will be validated, whereas with a dict they will not be.
            include_raw: If False then only the parsed structured output is returned. If
                an error occurs during model output parsing it will be raised. If True
                then both the raw model response (a BaseMessage) and the parsed model
                response will be returned. If an error occurs during output parsing it
                will be caught and returned as well. The final output is always a dict
                with keys "raw", "parsed", and "parsing_error".

        Returns:
            A Runnable that takes any ChatModel input and returns as output:

                If include_raw is True then a dict with keys:
                    raw: BaseMessage
                    parsed: Optional[_DictOrPydantic]
                    parsing_error: Optional[BaseException]
If include_raw is False then just _DictOrPydantic is returned,
                where _DictOrPydantic depends on the schema:

                If schema is a Pydantic class then _DictOrPydantic is the Pydantic
                    class.

                If schema is a dict then _DictOrPydantic is a dict.

        Example: Pydantic schema (include_raw=False):
            .. code-block:: python

                from langchain_experimental.llms import OllamaFunctions
                from pydantic import BaseModel

                class AnswerWithJustification(BaseModel):
                    '''An answer to the user question along with justification for the answer.'''
                    answer: str
                    justification: str

                llm = OllamaFunctions(model="phi3", format="json", temperature=0)
                structured_llm = llm.with_structured_output(AnswerWithJustification)

                structured_llm.invoke("What weighs more a pound of bricks or a pound of feathers")

                # -> AnswerWithJustification(
                #     answer='They weigh the same',
                #     justification='Both a pound of bricks and a pound of feathers weigh one pound. The weight is the same, but the volume or density of the objects may differ.'
                # )

        Example: Pydantic schema (include_raw=True):
            .. code-block:: python

                from langchain_experimental.llms import OllamaFunctions
                from pydantic import BaseModel

                class AnswerWithJustification(BaseModel):
                    '''An answer to the user question along with justification for the answer.'''
                    answer: str
                    justification: str

                llm = OllamaFunctions(model="phi3", format="json", temperature=0)
                structured_llm = llm.with_structured_output(AnswerWithJustification, include_raw=True)

Therefore, the primary reason ChatOllama fails to produce the correct query output while ChatGroq succeeds is the lack of support for the tool_choice parameter in ChatOllama ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.