RAFT: (RAG + Fine Tuning) in Azure AI Studio: Session Resources

[multispark]

This is the resources and discussion thread for this live stream: https://aka.ms/raghack/raft
It will be updated with links after the session.

Recording: https://www.youtube.com/watch?v=UrfmlqHgZLA
Slides: PPT

Please ask any follow-up questions here, and the speakers will see them.

[DuboisABB]

It would be interesting to comment about RAFT vs GraphRAG. Cost, use cases that would be appropriate for each, etc. Perhaps a topic for a future session or blog post?

[cedricvidal]

Hey, excellent suggestion, thank you for suggesting this! I don’t know the specifics of GraphRAG but my first impression is that assuming that GraphRAG targets the way documents are indexed and retrieved to improve recall, you still need to feed the retrieved chunks to the LLM for in-context learning and generation and RAFT can be used to improve the precision of that final in-context learning and generation part like it does for normal RAG. Definitely an interesting subject to dig more into!

[DuboisABB]

Yes, I can see that RAFT and GraphRAG could be combined. Both approaches are fairly expensive upfront, either for the fine tuning or the graph generation. The combination would end up very expensive but could yield impressive results.

As a follow-up question, did you explore fine tuning a more advanced model like GPT 4o mini? This new model is so cheap that perhaps there is no point in using a less advanced (cheaper?) model anymore?

[cedricvidal]

Both approaches are fairly expensive upfront

I can't talk for the cost of GraphRAG but it's higher than normal RAG for sure. As for the cost of RAFT, it's actually very cheap when compared to full weight fine tuning, only about $100 to fine tune a 8B model in less than an hour.

Eventually, the model will drift and not stay relevant with the latest documents so it's need to periodically refresh the fine tuned model but still very reasonable for enterprise scenarios.

did you explore fine tuning a more advanced model like GPT 4o mini

I haven't tried yet. gpt-40-mini is indeed available for fine-tuning and indeed quite cheap, definitely something to try!

If you try and have success, feel free to submit a PR, it would be great having support for this model in the recipe repo.

[cedricvidal]

Funny timing, just after I replied to your message, I learned that @azeltov and @vhoudebine had published a lab applying RAFT to a banking scenario using 4o as teacher and 4o-mini as student.

You might want to go look at what they've done:
The blog post: https://github.com/azeltov/raft-blog
The repo: https://github.com/Azure-Samples/azure-openai-raft

[Ashish-Soni08]

If we have performed RAFT on a certain document and fine-tuned a model accordingly, do we still need to provide the original source document when generating citations for responses to user queries?

[cedricvidal]

Hey Ashish,

Very good question! Yes, you do. Let me explain why. RAFT on Azure AI uses LORA or Low Order Ranking Algorithm, which updates the weights of the top layer of the model, it doesn't do full weights fine-tuning, this is why it is so cheap and fast.

With the combination of RAFT and LORA, the model is not taught new fundamental knowledge, indeed it becomes an expert at understanding the domain, focusing its attention on the citations that are the most useful to answer a question but it doesn't contain all the information about the domain. It is like the librarian analogy I used during the session (see video), a librarian doesn't know the content of all the books perfactly, but it knows which books contains the answers to a given question.

Another way to look at it is from a standpoint of information theory. Because LoRA only updates the top layer, there is only so much information you can store in that thin top layer as opposed to full weight fine tuning which updates all the weight bottom to top of the model.

LoRA might look like a limitation but it's actually perfect when used in combination with RAFT and RAG. You get the best of RAG and fine-tuning. RAG provides access to a potentially infinite amount of information and RAFT on LoRA provides a model which is an expert at understanding the documents retrieved by RAG at a fraction of the cost of full weight fine-tuning. And even with full weight fine-tuning, a model could still hallucinate, grounding is mandatory to get accurate answers.

[Ashish-Soni08]

Thank you so much for the great explanation, Cedric :)

[cedricvidal]

You’re welcome!