Structured input (JSON) with example selector

[robsyc]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

Prompt templates and example selector


ANNOTATOR_EXAMPLE_PROMPT = PromptTemplate(
    input_variables=["input", "output"],
    template="# Input:\n\n\n{input}```\n\n# Output:\n\n```{output}```"
)

ANNOTATOR_EXAMPLES = [
    {
        "input": """{"text":["Psoriasis is untreated (as ...""", 
        "output": """ResolvedCriterion(groups=['Psoriatic group'], criter..."""
    },
    # {
    # 	 "input": """This is a test""", # commenting out the previous example and uncommenting this example resolves the error
    # 	 "output": """ResolvedCriterion(groups=['Psoriatic group'], criter..."""
    # }
]

ANNOTATOR_EXAMPLE_SELECTOR = MaxMarginalRelevanceExampleSelector.from_examples(
    ANNOTATOR_EXAMPLES,
    emb,
    FAISS,
    k=8  # Number of examples to include (this can go up to 8 because the examples aren't too long)
)

ANNOTATOR_EXAMPLES_PROMPT = FewShotPromptTemplate(
    example_selector=ANNOTATOR_EXAMPLE_SELECTOR,
    example_prompt=ANNOTATOR_EXAMPLE_PROMPT,
    prefix="\n\n---\n\n**EXAMPLES**",
    suffix="---\n\n",
    input_variables=["task"]
)

Input for the example selector (i.e. task)

criterion.model_dump_json()
# >>> '{"text":["Psoriasis is untreated ...}'

Line that gives the error

ANNOTATOR_EXAMPLES_PROMPT.format(task=criterion.model_dump_json())

KeyError: '"text"'

Description

Goal:

• LLM takes as input a pydantic object in json and returns new pydantic object (because directly passing pydantic object as input doesn't work)
• Improve performance by adding examples using example selector

Problem: When formatting the ANNOTATOR_EXAMPLES_PROMPT, the "input" field in the examples is not being treated as a regular string to be embedded and compared against the current input. Instead a KeyError is thrown. Based on what I found online I think it's somehow related to the curly brackets in the json input, however, using a r-string or placing escape \ before the brackets is not resolving the issue.

Larger context: Little documentation is given online on what best practices are for provided structured input to LLMs... Should I even be passing json as input to an LLM? Downstream, I add these examples to the system prompt message like this:

messages_annotator = [
    SystemMessage(content=ANNOTATOR_PROMPT.format(resources=resources) + ANNOTATOR_EXAMPLES_PROMPT.format(task=criterion.model_dump_json())),
    HumanMessage(content=criterion.model_dump_json()),
]
response_annotator = annotator_llm.invoke(messages_annotator)

But I'm not sure if this is really the way to go.

System Info

langchain==0.2.16
langchain-chroma==0.1.2
langchain-community==0.2.16
langchain-core==0.2.38
langchain-openai==0.1.21
langchain-text-splitters==0.2.2

Platform: Linux

Python version: Python 3.10.12

[robsyc]

As for the error, this can be resolved using criterion.model_dump_json().replace("{", "{{").replace("}", "}}")

But the main question remains: What are best practices for structured input?

[dosubot[bot]]

Hey, @robsyc! 👋 I'm here to help you with any bugs, questions, or contributions you need assistance with. Let's squash those bugs together!

I found a similar open issue that might be relevant to your problem: Error happened in PromptTemplate + PydanticOutputParser. The issue was closed, and the suggested solution involves attaching instructions after formatting the prompt template. Here's the code snippet provided:

def main():
    output_parser = PydanticOutputParser(pydantic_object=CommandResponse)
    instruction = output_parser.get_format_instructions()

    _prompt = """
    ## User Demand
    {user_input}

    ## Pending Control Task Queue
    {task_queue}
    """

    prompt = PromptTemplate(
        template=f"{_prompt}",
        input_variables=["user_input", "task_queue"],
    )

    _input = prompt.format_prompt(user_input="hello", task_queue="aaa")
    _input = f"{_input}\n{instruction}"
    print(_input)

However, this solution is not ideal as it may still have issues determining if a variable does not exist ^[1].

To resolve the KeyError: '"text"' when formatting the ANNOTATOR_EXAMPLES_PROMPT with a pydantic object in JSON for an LLM in LangChain, ensure that the schema dictionary includes the "text" key. This key is likely required by the prompt template or the LLM chain configuration.

Here's a snippet to illustrate how you can modify the schema dictionary to include the "text" key:

if isinstance(schema, type) and is_basemodel_subclass(schema):
    schema_dict = cast(dict, schema.schema())
else:
    schema_dict = cast(dict, schema)

# Ensure the schema_dict includes the "text" key
if "text" not in schema_dict:
    schema_dict["text"] = "Default text value or appropriate value"

function = {
    "name": schema_dict["title"],
    "description": schema_dict["description"],
    "parameters": schema_dict,
}

This modification ensures that the "text" key is present in the schema dictionary, which should resolve the KeyError ^[2].

To continue talking to Dosu, mention @dosu.