Pros and cons of using function calling with Open AI?

[JasonHaley]

I've been looking at alot of RAG demos this month and have noticed there seems to be two types of implementations integrating Open AI calls into the system:

1. do the search on our data first, build the detailed prompt from the results, then call open ai
2. build a prompt, call open ai and have it do the search on our data in a function (callback) pattern

Since I've still got so much to learn with this stuff, I was wondering if someone more experienced could give us the pros and cons to using the 2 approaches.

[pamelafox]

Hm, which demos are showing #2? Is that using the dataSources parameter? For #2, is it actually answering the question when doing the search, or making a second call to answer it?

Our approaches are described here:
https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/customization.md#chatask-tabs

I'm happy to compare the approaches but need to wrap my head around exactly what the #2 approach is.

[JasonHaley]

I've got a bunch of demos on my desktop, I'll look up the ones that use functions and get back to you tomorrow. When I mention functions (not sure if I'm clear) I mean Open AI Function calling & How to use function calling with Azure OpenAI Service (Preview)

[JasonHaley]

Ok, I other examples, but to focust the discussion it looks like its in the repo for the hacathon demo you point us at. For above I was referring to the approaches used in:

1. app/backend/approaches/retrievethenread.py
2. app/backend/approaches/chatreadretrieveread.py

Looks like the term now is tools (not functions)

I don't have alot of experience with python yet and don't see the similiar options for chat approaches in the C# version of the repo.

The first example I've seen it in C# (which may be deprecated now) was here: Azure OpenAI RAG Pattern using Functions and a SQL Vector Database but I also noticed it in the Ignite demo for eShop code that I downloaded in November. The same "FromFunctionCall" doesn't seem to be in the eShop main branch any more.

[pamelafox]

Okay, right, so retrievethenread.py only makes a single call to Chat Completion, to answer the question, while chatretretrieveread.py first makes an initial call to ChatCompletion using tools with a single function search_sources(query), then uses that query to do the search. That was a change by Sri (who you saw on today's stream) to improve the reliability of search results. It worked fairly well before, but the usage of tools forces the GPT to think more carefully about the structure of its result. We want it to give a good keyword search string, and it gives us just that and only that.

I think it's nice that we use tools there because that's also a great place for other developers to add additional function calls, like to search other sources or do more structured queries. There's an example here which calls GitHub issues: https://gist.github.com/pamelafox/a3fdea186b687509c02cb186ca203328#file-chatreadretrieveread-py-L124

Does that clarify?