[Feature Request]: Document Groq API support

[GitDakky]

Is your feature request related to a problem? Please describe.

AutoGen is too slow using OpenAi API and needs the speed of Groq LLM API

Describe the solution you'd like

I would like to be able to use the Groq API to run the agents, The API info is below:
import os

from groq import Groq

client = Groq(
api_key=os.environ.get("GROQ_API_KEY"),
)

chat_completion = client.chat.completions.create(
messages=[
{
"role": "user",
"content": "Explain the importance of fast language models",
}
],
model="mixtral-8x7b-32768",
)

print(chat_completion.choices[0].message.content)

Additional context

The speed of Groq with AutoGen would be fantastic

[ekzhu]

What's preventing you from setting the base_url, model, and api_key of your Groq AI API directly in the llm_config?

[Josephrp]

groq is a "drop in replacement" just configure it based on the above, and try it out

[roundrobbin]

i tryed it with base_url, model, and api_key in llm_config, but there are errors, the communication between agents fails. It runs fine with openai.
so the first agent generate code, but the second can not read....

I think I see what's going on! It seems like my previous response didn't go through. Thank you for pointing it out!

---

WARNING:autogen.agentchat.groupchat:GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:
It seems like we're having a bit of a communication breakdown! Don't worry, it happens to the best of us. If you're trying to say something, feel free to try again, and I'll do my best to help you out!

[roundrobbin]

it works now sometimes. With a very simple question until the end, but with this warning:
WARNING:autogen.agentchat.groupchat:GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:
Thank you for the feedback! I'm glad you found the response accurate and helpful. I agree that including explanations for the code can be very helpful for users who are not familiar with the library or programming language. I'll keep that in mind for future responses.

so with groq often there fails the connection between the agents somehow. tryed mixtral 8 and llama3 8.

[Nathan-Intergral]

I am also testing out groq as a replacement for gpt at the moment. I have tested all the different models available with groq and it has resulted in this error in most cases

openai.BadRequestError: Error code: 400 - {'error': {'message': "Failed to call a function. Please adjust your prompt. See 'failed_generation' for more details.", 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation':

Groq supports function calling but perhaps the way I am defining the functions with autogen is causing a problem? I have had to switch around my code to use Langchain's tool classes for function declaration as it was the only way i could resolve the problem i was having of passing a context around the tools. I believe there is a difference between tools and functions in autogen if i am correct?

Does anyone have any idea how i might resolve this? Also has anyone tried llama 3 70b?

[GitDakky]

@roundrobbin - I think it's because Groq is so fast and they have substantially reduced the token limits. It could be the case that there may be a mismatch between the self imposed rate limiting (that might or might not be baked in to AutoGen) and the capabilities of Groq

[WebsheetPlugin]

I am also testing out groq as a replacement for gpt at the moment. I have tested all the different models available with groq and it has resulted in this error in most cases

openai.BadRequestError: Error code: 400 - {'error': {'message': "Failed to call a function. Please adjust your prompt. See 'failed_generation' for more details.", 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation':

Groq supports function calling but perhaps the way I am defining the functions with autogen is causing a problem? I have had to switch around my code to use Langchain's tool classes for function declaration as it was the only way i could resolve the problem i was having of passing a context around the tools. I believe there is a difference between tools and functions in autogen if i am correct?

Does anyone have any idea how i might resolve this? Also has anyone tried llama 3 70b?

That Groq supports function calling does not mean that the llama 3 70b model supports it... Are you sure what you are trying to do is valid? I plan also to use Groq with llama 3 70b, but without function calling.

You should debug the request it makes for both (Langchain and Autogen) and compare what the difference is.

Without Autogen does the function calling work for you with llama 3 70b?

Regarding your question, in Openai functions have been deprecated, and tools are the way forward. The main difference I am aware is that tools can be selected multiple times and that you can force the model to use a tool.

[Nathan-Intergral]

I am also testing out groq as a replacement for gpt at the moment. I have tested all the different models available with groq and it has resulted in this error in most cases
openai.BadRequestError: Error code: 400 - {'error': {'message': "Failed to call a function. Please adjust your prompt. See 'failed_generation' for more details.", 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation':
Groq supports function calling but perhaps the way I am defining the functions with autogen is causing a problem? I have had to switch around my code to use Langchain's tool classes for function declaration as it was the only way i could resolve the problem i was having of passing a context around the tools. I believe there is a difference between tools and functions in autogen if i am correct?
Does anyone have any idea how i might resolve this? Also has anyone tried llama 3 70b?

That Groq supports function calling does not mean that the llama 3 70b model supports it... Are you sure what you are trying to do is valid? I plan also to use Groq with llama 3 70b, but without function calling.

You should debug the request it makes for both (Langchain and Autogen) and compare what the difference is.

Without Autogen does the function calling work for you with llama 3 70b?

Regarding your question, in Openai functions have been deprecated, and tools are the way forward. The main difference I am aware is that tools can be selected multiple times and that you can force the model to use a tool.

@WebsheetPlugin I have come back today and tested groqs llama 3 70b model with autogen and it now works, even with functions so good news!

I was not aware that functions have been deprecated, I currently rely on them as i found no way with tools to provide any sort of auth values or anything to the tools for api calls. Using langchain tool classes and providing them as functions allows me to init the langchain tool with any variables i want, api keys etc

Its interesting you mention the difference with being able to force the model to use a tool. I am seeing this issue currently with my testing of llama 3 70b, its not using the functions when it needs to be and instead hallucinates

[523550914]

I am also testing out groq as a replacement for gpt at the moment. I have tested all the different models available with groq and it has resulted in this error in most cases
openai.BadRequestError: Error code: 400 - {'error': {'message': "Failed to call a function. Please adjust your prompt. See 'failed_generation' for more details.", 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation':
Groq supports function calling but perhaps the way I am defining the functions with autogen is causing a problem? I have had to switch around my code to use Langchain's tool classes for function declaration as it was the only way i could resolve the problem i was having of passing a context around the tools. I believe there is a difference between tools and functions in autogen if i am correct?
Does anyone have any idea how i might resolve this? Also has anyone tried llama 3 70b?

That Groq supports function calling does not mean that the llama 3 70b model supports it... Are you sure what you are trying to do is valid? I plan also to use Groq with llama 3 70b, but without function calling.
You should debug the request it makes for both (Langchain and Autogen) and compare what the difference is.
Without Autogen does the function calling work for you with llama 3 70b?
Regarding your question, in Openai functions have been deprecated, and tools are the way forward. The main difference I am aware is that tools can be selected multiple times and that you can force the model to use a tool.

@WebsheetPlugin I have come back today and tested groqs llama 3 70b model with autogen and it now works, even with functions so good news!

I was not aware that functions have been deprecated, I currently rely on them as i found no way with tools to provide any sort of auth values or anything to the tools for api calls. Using langchain tool classes and providing them as functions allows me to init the langchain tool with any variables i want, api keys etc

Its interesting you mention the difference with being able to force the model to use a tool. I am seeing this issue currently with my testing of llama 3 70b, its not using the functions when it needs to be and instead hallucinates

hi, can I ask you how to change to use llama 3 70b in Autogen with groq
/

[523550914]

can everybody share the base_url,I have try for this one"https://api.groq.com/openai/v1/models",but it doesn't work

[WebsheetPlugin]

Here you go:
config_list = [{"model": "llama3-70b-8192", "api_key": "gsk_XXX", "base_url":"https://api.groq.com/openai/v1"}]

and later:
llm_config = {
"temperature": 0,
"config_list": config_list,
}

engineer = autogen.AssistantAgent(
    name="Engineer",
    llm_config=llm_config,
    system_message="""
    I'm Engineer. I'm expert in python programming. I'm executing code tasks required by Admin.
    """,
)

[ekzhu]

@WebsheetPlugin can you add this to our documentation? https://microsoft.github.io/autogen/docs/topics/non-openai-models/about-using-nonopenai-models, and
a new markdown page under /website/docs/topics/non-openai-models/

[523550914]

Here you go: config_list = [{"model": "llama3-70b-8192", "api_key": "gsk_XXX", "base_url":"https://api.groq.com/openai/v1"}]

and later: llm_config = { "temperature": 0, "config_list": config_list, }

engineer = autogen.AssistantAgent(
    name="Engineer",
    llm_config=llm_config,
    system_message="""
    I'm Engineer. I'm expert in python programming. I'm executing code tasks required by Admin.
    """,
)

Thank you so much! I have another question that how do you make sure that it uses the groq?

[ekzhu]

Thank you so much! I have another question that how do you make sure that it uses the groq?

It must be using Groq if there is only one entry in the config list.

[WebsheetPlugin]

You can use gather_usage_summary

from autogen.agentchat.utils import gather_usage_summary

usage_summary = gather_usage_summary(engineer)
usage_summary = usage_summary["usage_excluding_cached_inference"]
print("usage_summary:", usage_summary)

Result:
usage_summary: {'total_cost': 0.000438, 'llama3-70b-8192': {'cost': 0, 'prompt_tokens': 907, 'completion_tokens': 255, 'total_tokens': 1162}, 'gpt-3.5-turbo-0125': {'cost': 0.000438, 'prompt_tokens': 738, 'completion_tokens': 46, 'total_tokens': 784}}

Here you can see that llama3-70b-8192 has been used.

[WebsheetPlugin]

@ekzhu I have never added something to an

@WebsheetPlugin can you add this to our documentation? https://microsoft.github.io/autogen/docs/topics/non-openai-models/about-using-nonopenai-models, and a new markdown page under /website/docs/topics/non-openai-models/

I have never added something to a documentation git. But I would like to do it, so I can contribute.

Please just confirm:

1. in https://microsoft.github.io/autogen/docs/topics/non-openai-models/about-using-nonopenai-models, I add information about groq into the doc. I simply add Groq with a hyperlink to Groq (A). and below to the example (to new subpage - B).
   See Attached Image for A and B.
   

2. I create a new page, dedicated to Grooq (/website/docs/topics/non-openai-models/grooq). Here I showcase the example I provided earlier on how to set it up.

[ekzhu]

Sounds good! Let's do it.

[jtoy]

Groq isn't really an option now because you cannot get over 30 calls a minute and most agents will make many calls so I would be careful about using too much groq now unless you are certain you can keep the call count low.

[Gal-Gilor]

Groq isn't really an option now because you cannot get over 30 calls a minute and most agents will make many calls so I would be careful about using too much groq now unless you are certain you can keep the call count low.

Groq’s client provides an optional retry/max_retries parameter. Unfortunately, adding those to the config increases the chance of a the “Extract_response” from {response} is None”. Warning

it will require a separate issue, but can we enable retries for Groq?

[exp7l]

What should be the setting for llm_config to trigger tool use?

def list_files(path: str) -> List[str]:
    return [f for f in os.listdir(path)]


spec_writer.register_for_llm(name="list_files", description="Lists files and folders")(fs.list_files)

> list ./

Do you know why list_files is not called? Thanks.

[jiveshkalra]

I tried using the given llm_config but it doesnt seem to work anymore?

config_list = [{
        "model": "llama3-70b-8192", 
        "api_key": "gsk_***", 
        "base_url": "https://api.groq.com/openai/v1"
    }]

when running autogen with these configurations , I am getting the following error

  File "D:\Anaconda3\envs\pyautogen\lib\site-packages\openai\api_requestor.py", line 775, in _interpret_response_line
    raise self.handle_error_response(
openai.error.AuthenticationError: Incorrect API key provided: gsk_WhpK********************************************EetK. You can find your API key at https://platform.openai.com/account/api-keys.

I dont know why this is happening , I have set the base url to groq , still its taking base url as openai

if anyone has any clue why this is happening , any help will be much appreciated ,
Thank you

[Gal-Gilor]

I tried using the given llm_config but it doesnt seem to work anymore?

config_list = [{
        "model": "llama3-70b-8192", 
        "api_key": "gsk_***", 
        "base_url": "https://api.groq.com/openai/v1"
    }]

when running autogen with these configurations , I am getting the following error

  File "D:\Anaconda3\envs\pyautogen\lib\site-packages\openai\api_requestor.py", line 775, in _interpret_response_line
    raise self.handle_error_response(
openai.error.AuthenticationError: Incorrect API key provided: gsk_WhpK********************************************EetK. You can find your API key at https://platform.openai.com/account/api-keys.

I dont know why this is happening , I have set the base url to groq , still its taking base url as openai

if anyone has any clue why this is happening , any help will be much appreciated , Thank you

Which autogen version are you using?
For good measure, could you try regenerating the api key?

[jiveshkalra]

Version -> 0.1.14

I also tried regenerating the API key, but still same issue
it seems like the autogen didnt take the groq's base url and instead is always taking openai's url to authenticate / use llms

[ghost]

These are hackers hacking me please contact fbi

…

On Fri, Jun 14, 2024, 12:13 AM Jivesh Kalra ***@***.***> wrote: Version -> 0.1.14 I also tried regenerating the API key, but still same issue it seems like the autogen didnt take the groq's base url and instead is always taking openai's url to authenticate / use llms — Reply to this email directly, view it on GitHub <#2555 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BI3MSTDVHOFZ5ISGVWMF2BDZHJ3Y3AVCNFSM6AAAAABHBGMDH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRXGIZDKMJSHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[tarvindersi]

I am curious if anyone has been able to make Autogen work with Groq API -- in particular, their experience with function calling and tools.

Thank you.

[Gal-Gilor]

@tarvindersi, I got it to work. The above configuration worked for me.
I had a positive experience going through the agent planning notebook.

[cbarkinozer]

I just tried using llama3-70b from Groq with Autogen. The library you need to install is "pyautogen" not "autogen" when using Python.
The below configuration worked for me (I did not thoroughly test it, I was just able to run a simple example):

GROQ_API_KEY = os.getenv("GROQ_API_KEY")
prompt_price_per_1k=0
completion_token_price_per_1k=0
config_list= [{"model": "llama3-70b-8192", "api_key": GROQ_API_KEY, "base_url": "https://api.groq.com/openai/v1", "price": [prompt_price_per_1k, completion_token_price_per_1k], "frequency_penalty": 0.5, "max_tokens": 2048, "presence_penalty": 0.2, "temperature": 0.5, "top_p": 0.2}]
llm_config = {"config_list":config_list}

Note: The price field is to silence warnings about price calculation.