Internationalization for RAG apps: Session Resources

[multispark]

This is the resources and discussion thread for this live stream: https://aka.ms/raghack/language
It will be updated with links after the session.

Recording: https://www.youtube.com/watch?v=GWb6fICZWZY

Please ask any follow-up questions here, and the speakers will see them.

[tonybaloney]

Links:

• Tokenizer UI
• Blog post on CJK and the token ratio
• LangChain Token based recursive splitting
• Service Availability Tables
• Source Code for Trading Website

[tonybaloney]

The code used for the query prep was:

completion = client.chat.completions.create(
        model=completions_deployment,
        messages= [
        {
            "role": "system",
            "content": 
            """  
                Generate a full-text search query for a SQL database based on a user query. 
                Do not generate the whole SQL query; only generate string to go inside the MATCH parameter for FTS5 indexes. 
                Use SQL boolean operators if the user has been specific about what they want to exclude in the search.
                If the query is not in English, always translate the query to English.
                If you cannot generate a search query, return just the number 0.
            """
        },
        {   "role": "user",
            "content": f"Generate a search query for: A really nice winter jacket"
        },
        {  "role": "assistant",
            "content": "winter jacket"
        },
        {   "role": "user",
            "content": "Generate a search query for: 夏のドレス"
        },
        {   "role": "assistant",
            "content": "summer dress"
        },
        {
            "role": "user",
            "content": f"Generate a search query for: {query}"
        }],
        max_tokens=100, # maximum number of tokens to generate
        n=1, # return only one completion
        stop=None, # stop at the end of the completion
        temperature=0.3, # more predictable
        stream=False, # return the completion as a single string
        seed=1, # seed for reproducibility
    )
    

[KhaoticMind]

Hi @tonybaloney one doubt. In the case were you need to index multiple documents from different languages (pt-BR and en-US in my case), and the user is, most probably a pt-BR speaker, whats the best approach? Using text-3-large is a good start, as pointed out in your presentation, but what else can we Do?

Should we instruct the LLM to "think" in english (like to generate the search query)? Translate the documents before vectorizing? Store the language of the document on the database to full text search? Remembering the the output to the user will have to be in the same language he asked the question (which I know might be another issue since some RAG docs will be in english and this might cofuse the LLM).

Thanks for the great session! Really got me thinking :)

[pamelafox]

Just sharing some of my own thoughts, as Anthony's timezone won't wake up til later .In your case, do you know if all your users will be PT-BR speakers? Or will some be English speakers?
If they were all PT-BR speakers, I think you would want to write the system prompt in PT-BR and provide few shot examples of responding in Portuguese. That would hopefully persuade it to answer in PT-BR, regardless of the document source language.

As for searching, you shouldn't technically need to translate the documents before vectorizing, since embedding models can help compensate for different languages of query versus sources, but you could consider that if you're seeing poor retrieval quality. I wouldn't bother with it until I could see clearly that there were retrieval issues.

[KhaoticMind]

Great suggestions Pamela! I'm not really giving the few shots examples the importance I should.
Most of my users will probably ask in PT-BR (that's where I'm and most of my users are), but some might want to speak in english because it might make more sense for them (thinking in IT and Cybersecurity use cases, where most of literature is in english). I'll try your suggestion later and see how it works.

As for searching, the embedding will probably take cara of most issues yes, I'm worrying before seeing problems. I did a quick experiment with prompt engineering and the line bellow worked as I'd expect.
"Always use the tool twice: with words in portuguese and in english to find related news (the database has both languages)."
The LLM used the tool (that simply do RAG on PGSQL) twice:

Arguments:
{"query":"inflação Brasil setembro 2024"}

And

Arguments:
{"query":"Brazil inflation September 2024"}

Indeed most of the results were similar (but with a little difference on the ordering). So yeah, right now I'm probably just being redundant.
Thanks for the insights again! Will keep a closer look at the evaluation session on thursday :)