Doesn’t Ollama already support OpenAI api?

[NeverOccurs]

Then why the need to wrap up Ollama models with LiteLLM?

[ekzhu]

@marklysze would you like to elaborate?

[marklysze]

Hey @NeverOccurs, yep, Ollama does have an OpenAI-compatible API that can be used directly with AutoGen. If you don't need Function/Tool Calling, then I'd say Ollama's API is the way to go (saves having to have LiteLLM).

LiteLLM supports OpenAI's Function/Tool calling API parts and, together with Ollama, you can use function calling with AutoGen.

We are looking to add a direct Ollama API example as well in the documentation.

[NeverOccurs]

Thanks, really appreciate it. Actually I've been thinking about it a few days ago that if local models support function calling, then I saw this release. Will dig into it further.

[marklysze]

Anytime, if you want to chat with others interested in local models, feel free to join AutoGen's Discord - #alt-models channel is available for just that.

I'll close the discussion for now but feel free to create another here or on Discord if you have any.

[shuaib7860]

Hi sorry to open this issue again,

But does this also mean trying to use the RetreiveUserProxyAgent is not suitable for llama models just yet either because we need to do some function calling when trying to do a RAG agent?

[marklysze]

Okay... If you're able to try using the Ollama PR code I linked to, it would be good to know if that works. That should be better at handling a mix of tool calling and normal text generation.

[shuaib7860]

Hey @marklysze, Apologies for the delay but I was travelling over the weekend.

I gave your Olama PR code example a try with the Jack and Emma ConversableAgents and I get the same warning message, although it does work.

I attach a screenshot just for your reference. Is anyone else seeing this warning too?

[shuaib7860]

Actually I have just thought of a potential reason why this warning message pops up but I'm not sure. So I'll share my thoughts and look forward to hearing yours in response.

The LLM is hosted on a remote server and therefore I am port forwarding to the remote server from my local machine to interact with the LLM. Could this be the reason why this particular warning comes up as it looks to run the LLM locally and then just interacts with the one on the remote machine once the port forwarding kicks in?

Also one more thing to add is with the RetrieveUserProxyAgent tool the warning says

"Model llama3:70b not found. Using cl100k_base encoding."

So a little different but in essence the same warning. But this warning can be traced back to the token_count_utils python file.

autogen/autogen/token_count_utils.py

Line 82 in 5b1dc3b

logger.warning(f"Model {model} not found. Using cl100k_base encoding.")

[marklysze]

Hey @shuaib7860, sorry I haven't been that available over the last few weeks.

If you're referring to the message "...The cost will be 0", this is related to the way that AutoGen calculates inference costs for you. You can add that price field to your your config to set the price as zero. See this comment for an example of how to do that.

For the base encoding warning, and thanks for the reference to the token_count_utils file, this is just so that tokens can be counted. If you're not concerned by token costs then this can be ignored. I'm not sure if there's a way to turn it off.

[shuaib7860]

Many thanks @marklysze for the price field suggestion. And no worries on the time it took to get back to me. Thank you for all your help.