Using Field Descriptions for Enhanced Data Narratives with LLMs - By Using Python REPL Tool

[mraguth]

Assume that an LLM is generating insights from a DataFrame. How can LLMs use 'field descriptions' to provide a narrative based on these descriptions rather than just the field names? For example, instead of saying '500 EMPID are present in the data,' I want the LLM to say '500 Employees are present in the data.'

I have a separate file with field descriptions, such as EMPID = Employee. My objective is for the LLM to recognize that EMPID corresponds to Employee and thus generate the final response as '500 Employees' instead of '500 EMPID.'

Thank you for your suggestions, and could you please provide sample code?

Sample Code:

import pandas as pd

# Sample DataFrame for employees
employee_data = {
    'EMPID': [101, 102, 103, 104, 105],
    'EMPName': ['Alice', 'Bob', 'Charlie', 'David', 'Eve']
}
employee_df = pd.DataFrame(employee_data)
employee_df.head()

insights_data = {
    'Field_Name': ['EMPID', 'EMPName'],
    'Insights to LLM': [
        'EMPID represents the unique identifier assigned to each employee in the dataset.',
        'EMPName denotes the full name of the employee as recorded in the dataset.'
    ]
}
insights_df = pd.DataFrame(insights_data)

insights_df.head()

Python Tool used:

repl = PythonAstREPLTool(
    locals={"df": df},
    name="python_repl",
    description="Runs code and returns the output of the final line",
    args_schema=PythonInputs,
)

[dosubot[bot]]

To achieve your goal of using 'field descriptions' to provide a narrative based on these descriptions rather than just the field names, you can follow these steps:

1. Load the field descriptions from the separate file.
2. Create a mapping between field names and their descriptions.
3. Use this mapping to generate more descriptive narratives when querying the DataFrame.

Here is a sample code to demonstrate this:

import pandas as pd

# Sample DataFrame for employees
employee_data = {
    'EMPID': [101, 102, 103, 104, 105],
    'EMPName': ['Alice', 'Bob', 'Charlie', 'David', 'Eve']
}
employee_df = pd.DataFrame(employee_data)

# Sample DataFrame for field descriptions
insights_data = {
    'Field_Name': ['EMPID', 'EMPName'],
    'Description': [
        'Employee',  # Description for EMPID
        'Employee Name'  # Description for EMPName
    ]
}
insights_df = pd.DataFrame(insights_data)

# Create a dictionary mapping field names to descriptions
field_desc_dict = dict(zip(insights_df['Field_Name'], insights_df['Description']))

# Function to generate narrative based on field descriptions
def generate_narrative(query_result, field_desc_dict):
    narrative = []
    for field, value in query_result.items():
        description = field_desc_dict.get(field, field)
        narrative.append(f"{description}: {value}")
    return "\n".join(narrative)

# Example query
query_result = employee_df.iloc[0].to_dict()

# Generate narrative
narrative = generate_narrative(query_result, field_desc_dict)
print(narrative)

In this example:

• insights_df contains the field descriptions.
• The generate_narrative function uses the field_desc_dict to replace field names with their descriptions in the query result.

This approach ensures that the generated insights are more descriptive and user-friendly.

For integrating this with the PythonAstREPLTool, you can set up the tool as follows:

from langchain.experimental.tools.python.tool import PythonAstREPLTool
from langchain.experimental.tools.python.inputs import PythonInputs

# Initialize the REPL tool with the DataFrame and field descriptions
repl = PythonAstREPLTool(
    locals={"employee_df": employee_df, "field_desc_dict": field_desc_dict, "generate_narrative": generate_narrative},
    name="python_repl",
    description="Runs code and returns the output of the final line",
    args_schema=PythonInputs,
)

# Example usage of the REPL tool
query = """
query_result = employee_df.iloc[0].to_dict()
narrative = generate_narrative(query_result, field_desc_dict)
narrative
"""
output = repl._run(query)
print(output)

This setup allows you to use the PythonAstREPLTool to execute code that generates narratives based on field descriptions.

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu Good suggestion... as a final step, I am using repl tool in my invoke method. But my initial script only look at one dataframe 'df' in the template. How I accomidate both and tune this one? can you hlep? I am little new for langchain. Thank you for tuning my code.

from langchain.agents import create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.agents import AgentExecutor
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_experimental.tools import PythonAstREPLTool
from langchain.agents import AgentExecutor, OpenAIFunctionsAgent

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_experimental.tools import PythonAstREPLTool
from langchain.agents import AgentExecutor, OpenAIFunctionsAgent


class PythonInputs(BaseModel):
    query: str = Field(description="code snippet to run")


template = """
You are working with a pandas dataframe in Python. The name of the dataframe is `df`.
It is important to understand the attributes of the dataframe before working with it. This is the result of running `df.head().to_markdown()`

<df>
{dhead}
</df>

You are not meant to use only these rows to answer questions - they are meant as a way of telling you about the shape and schema of the dataframe.
You also do not have use only the information here to answer questions - you can run intermediate queries to do exploratory data analysis to give you more information as needed.
Convert your curiosity or questions into executable Python code using Pandas to learn as needed.
You can read and write files from the local computer drive.
You can build reports based on human requests, using field names from human responses.
As a Python-based AI agent, you perform comprehensive data analysis tasks.

"""  # noqa: E501

template = template.format(dhead=df.head().to_markdown())

# Define the prompt
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", template),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
        ("human", "{input}"),
    ]
)


# Initialize the REPL tool with the DataFrame and field descriptions
repl = PythonAstREPLTool(
    locals={"employee_df": employee_df, "field_desc_dict": field_desc_dict, "generate_narrative": generate_narrative},
    name="python_repl",
    description="Runs code and returns the output of the final line",
    args_schema=PythonInputs,
)
tools_repl = [repl]

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

# Define the prompt for agent2
agent1_prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a helpful assistant you use python REPL tools to do complex computations as well as narration"
        ),
        MessagesPlaceholder("chat_history", optional=True),
        ("human", "{input}"),
        MessagesPlaceholder("agent_scratchpad"),
    ]
)

agent1 = OpenAIFunctionsAgent(
    llm=llm, prompt=prompt, tools=tools_repl, agent1_prompt=agent1_prompt
)

agent_executor = AgentExecutor(
    agent=agent1,
    tools=tools_repl,
    max_iterations=10,
    early_stopping_method="generate",
    verbose=True,
    return_intermediate_steps=True,
) | (lambda x: x["output"])


# Typing for playground inputs
class AgentInputs(BaseModel):
    input: str


agent_executor = agent_executor.with_types(input_type=AgentInputs)
result = agent_executor.invoke({"input":"explain the dataset"})

[dosubot[bot]]

To modify your script to accommodate both DataFrames and tune it for use with the PythonAstREPLTool in LangChain, you need to update your imports to use the langchain_experimental package as the PythonAstREPLTool has been moved there. Here is an example of how you can modify your script:

1. Update the Template: Modify the template to include both DataFrames.
2. Initialize the REPL Tool: Include both DataFrames in the locals dictionary.
3. Adjust the Prompt and Agent: Ensure the prompt and agent are set up to handle multiple DataFrames.

Here is the updated script:

import pandas as pd
from langchain.agents import create_tool_calling_agent, AgentExecutor, OpenAIFunctionsAgent
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_experimental.tools.python.tool import PythonAstREPLTool

# Sample DataFrames for employees and insights
employee_data = {
    'EMPID': [101, 102, 103, 104, 105],
    'EMPName': ['Alice', 'Bob', 'Charlie', 'David', 'Eve']
}
employee_df = pd.DataFrame(employee_data)

insights_data = {
    'Field_Name': ['EMPID', 'EMPName'],
    'Description': [
        'Employee',  # Description for EMPID
        'Employee Name'  # Description for EMPName
    ]
}
insights_df = pd.DataFrame(insights_data)

# Create a dictionary mapping field names to descriptions
field_desc_dict = dict(zip(insights_df['Field_Name'], insights_df['Description']))

# Function to generate narrative based on field descriptions
def generate_narrative(query_result, field_desc_dict):
    narrative = []
    for field, value in query_result.items():
        description = field_desc_dict.get(field, field)
        narrative.append(f"{description}: {value}")
    return "\n".join(narrative)

# Define the template
template = """
You are working with two pandas dataframes in Python. The names of the dataframes are `employee_df` and `insights_df`.
It is important to understand the attributes of the dataframes before working with them. This is the result of running `employee_df.head().to_markdown()` and `insights_df.head().to_markdown()`:

<employee_df>
{employee_dhead}
</employee_df>

<insights_df>
{insights_dhead}
</insights_df>

You are not meant to use only these rows to answer questions - they are meant as a way of telling you about the shape and schema of the dataframes.
You also do not have to use only the information here to answer questions - you can run intermediate queries to do exploratory data analysis to give you more information as needed.
Convert your curiosity or questions into executable Python code using Pandas to learn as needed.
You can read and write files from the local computer drive.
You can build reports based on human requests, using field names from human responses.
As a Python-based AI agent, you perform comprehensive data analysis tasks.
"""  # noqa: E501

template = template.format(
    employee_dhead=employee_df.head().to_markdown(),
    insights_dhead=insights_df.head().to_markdown()
)

# Define the prompt
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", template),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
        ("human", "{input}"),
    ]
)

# Initialize the REPL tool with the DataFrames and field descriptions
repl = PythonAstREPLTool(
    locals={"employee_df": employee_df, "insights_df": insights_df, "field_desc_dict": field_desc_dict, "generate_narrative": generate_narrative},
    name="python_repl",
    description="Runs code and returns the output of the final line",
    args_schema=PythonInputs,
)
tools_repl = [repl]

# Define the prompt for agent1
agent1_prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a helpful assistant who uses Python REPL tools to do complex computations as well as narration."
        ),
        MessagesPlaceholder("chat_history", optional=True),
        ("human", "{input}"),
        MessagesPlaceholder("agent_scratchpad"),
    ]
)

# Create the OpenAIFunctionsAgent
agent1 = OpenAIFunctionsAgent(
    llm=llm, prompt=prompt, tools=tools_repl, agent1_prompt=agent1_prompt
)

# Create the AgentExecutor
agent_executor = AgentExecutor(
    agent=agent1,
    tools=tools_repl,
    max_iterations=10,
    early_stopping_method="generate",
    verbose=True,
    return_intermediate_steps=True,
) | (lambda x: x["output"])

# Typing for playground inputs
class AgentInputs(BaseModel):
    input: str

agent_executor = agent_executor.with_types(input_type=AgentInputs)
result = agent_executor.invoke({"input": "explain the dataset"})
print(result)

This script initializes the PythonAstREPLTool with both DataFrames and sets up the prompt to include information about both DataFrames. The AgentExecutor is then used to invoke the agent with a query. This setup ensures that the agent can handle multiple DataFrames and generate narratives based on field descriptions ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu...perfect! I will play around with more advanced examples and let you know.

[mraguth]

@dosu, I am using a few-shot template as shown below. My request is: Is it possible to integrate this into my 'agent_executor'? By doing so, I can run everything together before invoking. Please help.

https://python.langchain.com/docs/how_to/few_shot_examples/ (reference)

from langchain_core.prompts import FewShotPromptTemplate

prompt = FewShotPromptTemplate(
    examples=examples,
    example_prompt=example_prompt,
    suffix="Question: {input}",
    input_variables=["input"],

)

print(
    prompt.invoke({"input": "Who was the father of Mary Ball Washington?"}).to_string()
)

[dosubot[bot]]

Yes, it is possible to integrate the FewShotPromptTemplate into your agent_executor so that you can run everything together before invoking. You can define a prompt with a placeholder for the agent's scratchpad and then create an agent using this prompt. Here is an example:

from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate, FewShotChatMessagePromptTemplate

# Define few-shot examples
examples = [
    {"input": "2+2", "output": "4"},
    {"input": "2+3", "output": "5"},
]

# Create a prompt template for the examples
example_prompt = ChatPromptTemplate.from_messages(
    [
        ("human", "{input}"),
        ("ai", "{output}"),
    ]
)

# Create the few-shot prompt template
few_shot_prompt = FewShotChatMessagePromptTemplate(
    input_variables=["input"],
    example_prompt=example_prompt,
    examples=examples,
)

# Assemble the final prompt
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        few_shot_prompt,
        ("human", "{input}"),
        ("placeholder", "{agent_scratchpad}"),
    ]
)

# Define which tools the agent can use to answer user queries
tools = [
    Tool(
        name="ItemLookup",
        func=(lambda x: vocab_lookup(x, entity_type="item")),
        description="useful for when you need to know the q-number for an item",
    ),
    Tool(
        name="PropertyLookup",
        func=(lambda x: vocab_lookup(x, entity_type="property")),
        description="useful for when you need to know the p-number for a property",
    ),
    Tool(
        name="SparqlQueryRunner",
        func=run_sparql,
        description="useful for getting results from a wikibase",
    ),
]

# Create the agent and agent executor
agent = create_tool_calling_agent(model, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools)

# Invoke the agent executor with a query
agent_executor.invoke({"input": query})

This setup allows you to integrate the FewShotPromptTemplate into your agent_executor and run everything together before invoking. The few-shot examples help guide the model's responses, potentially improving its performance. The FewShotPromptTemplate can automatically infer input variables if not explicitly provided, making it flexible and easy to use. Additionally, the FewShotPromptTemplate supports both synchronous and asynchronous formatting of prompts, allowing for dynamic selection of examples based on the input ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu My goal is ingest few-shot prompt in the original agent_executor. Not to create separate agent_executor. Please help to converge in below agent executor.

# Create the AgentExecutor
agent_executor = AgentExecutor(
    agent=agent1,
    tools=tools_repl,
    max_iterations=10,
    early_stopping_method="generate",
    verbose=True,
    return_intermediate_steps=True,
) | (lambda x: x["output"])

# Typing for playground inputs
class AgentInputs(BaseModel):
    input: str

agent_executor = agent_executor.with_types(input_type=AgentInputs)
result = agent_executor.invoke({"input": "explain the dataset"})
print(result)

[dosubot[bot]]

To integrate a few-shot prompt into your existing agent_executor without creating a separate agent_executor, you can follow these steps:

1. Define the few-shot examples:

   examples = [
       {"input": "2+2", "output": "4"},
       {"input": "2+3", "output": "5"},
   ]

2. Create the example prompt template:

   from langchain_core.prompts import ChatPromptTemplate, FewShotChatMessagePromptTemplate
   
   example_prompt = ChatPromptTemplate.from_messages(
       [
           ("human", "{input}"),
           ("ai", "{output}"),
       ]
   )

3. Assemble the few-shot prompt template:

   few_shot_prompt = FewShotChatMessagePromptTemplate(
       examples=examples,
       example_prompt=example_prompt,
   )

4. Create the final prompt template:

   final_prompt = ChatPromptTemplate.from_messages(
       [
           ("system", "You are a helpful assistant."),
           few_shot_prompt,
           ("human", "{input}"),
       ]
   )

5. Integrate the final prompt template with your existing agent_executor:

   Assuming agent_executor is an instance of a chain that can accept a prompt template, you can directly use the final_prompt with it.

   from langchain.agents import AgentExecutor, OpenAIFunctionsAgent
   from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
   from langchain_experimental.tools.python.tool import PythonAstREPLTool
   from langchain_core.pydantic_v1 import BaseModel, Field
   
   # Define the prompt for agent1
   agent1_prompt = ChatPromptTemplate.from_messages(
       [
           ("system", "You are a helpful assistant who uses Python REPL tools to do complex computations as well as narration."),
           MessagesPlaceholder("chat_history", optional=True),
           ("human", "{input}"),
           MessagesPlaceholder("agent_scratchpad"),
       ]
   )
   
   # Create the OpenAIFunctionsAgent
   agent1 = OpenAIFunctionsAgent(
       llm=llm, prompt=final_prompt, tools=tools_repl, agent1_prompt=agent1_prompt
   )
   
   # Create the AgentExecutor
   agent_executor = AgentExecutor(
       agent=agent1,
       tools=tools_repl,
       max_iterations=10,
       early_stopping_method="generate",
       verbose=True,
       return_intermediate_steps=True,
   ) | (lambda x: x["output"])
   
   # Typing for playground inputs
   class AgentInputs(BaseModel):
       input: str
   
   agent_executor = agent_executor.with_types(input_type=AgentInputs)
   result = agent_executor.invoke({"input": "explain the dataset"})
   print(result)

By following these steps, you integrate the few-shot prompt into your existing agent_executor without creating a separate one. The FewShotChatMessagePromptTemplate class from LangChain allows you to dynamically select examples or use a fixed list, making it flexible for various use cases ^[1][2].

To continue talking to Dosu, mention @dosu.