-
Notifications
You must be signed in to change notification settings - Fork 9.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add chatml fallback for cpp llama_chat_apply_template
#8160
Add chatml fallback for cpp llama_chat_apply_template
#8160
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DeepSeek-V2-Lite no longer crashes with these fixes, but I think there's one unnecessary llama_chat_apply_template()
call that can be removed.
common/common.cpp
Outdated
if (fallback) { | ||
res = llama_chat_apply_template(nullptr, "chatml", chat.data(), chat.size(), add_ass, buf.data(), buf.size()); | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this really necessary? I mean you already called llama_chat_apply_template(nullptr, "chatml", ...)
in else
above, so why do you call it again?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah that's right, I forgot to delete the line inside else
condition
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I deleted this line of code (keep the line in else
branch). I've tested it once more time and confirmed that it's still working.
This will be merged once CI passed.
* add chatml fallback for cpp `llama_chat_apply_template` * remove redundant code
* add chatml fallback for cpp `llama_chat_apply_template` * remove redundant code
* add chatml fallback for cpp `llama_chat_apply_template` * remove redundant code
* add chatml fallback for cpp `llama_chat_apply_template` * remove redundant code
Fix problem with DeepSeek V2 chat model: #8068 (comment) (cc @fairydreaming )
DeepSeek is no longer crash now, but it's using chatml (not ideal). We can add template for deepseek later.
Demo: