Add chat template support for llama-cli #8068
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ngxson
added
the
Review Complexity : Low
This was referenced Jun 23, 2024
ggerganov
approved these changes
Jun 24, 2024
| std::string user_inp = params.conversation | ||
| ? chat_add_and_format("user", buffer) | ||
| : buffer; | ||
| // TODO: one inconvenient of current chat template implementation is that we can't distinguish between user input and special tokens (prefix/postfix) |
There was a problem hiding this comment.
When params.conversation == false there is an extra string copy that should be avoided here
Regarding the comment - can you illustrate with an example as I'm not sure what is the issue
There was a problem hiding this comment.
An example would be a prompt like this: Which one is correct HTML tag? <s> or <a>?
Some models having <s> as BOS will see the prompt as Which one is correct HTML tag? BOS or <a>?
Leaving special == false will fix that, but will also break chat template since we're now adding special tokens to user's text. This could be avoided with some more code. But IMO it's not really a big deal though, assuming that special tokens are unlikely to accidentally appear in the text.
There was a problem hiding this comment.
I added a std::move(buffer) since we no longer use buffer after this line. Is it OK to do so?
There was a problem hiding this comment.
Aha got it. Yes, for now let's make have the simple solution
ngxson
commented
Jun 24, 2024
ggerganov
reviewed
Jun 25, 2024
Co-authored-by: Georgi Gerganov <[email protected]>
ngxson
added
the
merge ready
|
It looks like it broke some models, here is the llama-cli output and brief gdb inspection from DeepSeek-V2-Lite: |
|
@fairydreaming The default behavior should be "if built-in template is not supported, we use chatml as fallback value" Turns out it's not the case here (I missed something). I'll need to push a fix for this. |
This was referenced Jun 27, 2024
arthw
pushed a commit
to arthw/llama.cpp
that referenced
this pull request
Jun 30, 2024
* add chat template support for llama-cli * add help message * server: simplify format_chat * more consistent naming * improve * add llama_chat_format_example * fix server * code style * code style * Update examples/main/main.cpp Co-authored-by: Georgi Gerganov <[email protected]> --------- Co-authored-by: Georgi Gerganov <[email protected]>
MagnusS0
pushed a commit
to MagnusS0/llama.cpp-normistral-tokenizer
that referenced
this pull request
Jul 1, 2024
* add chat template support for llama-cli * add help message * server: simplify format_chat * more consistent naming * improve * add llama_chat_format_example * fix server * code style * code style * Update examples/main/main.cpp Co-authored-by: Georgi Gerganov <[email protected]> --------- Co-authored-by: Georgi Gerganov <[email protected]>
ggerganov
added a commit
that referenced
this pull request
Jul 25, 2024
arthw
pushed a commit
to arthw/llama.cpp
that referenced
this pull request
Jul 27, 2024
mishig25
pushed a commit
to huggingface/huggingface.js
that referenced
this pull request
Aug 7, 2024
In this PR, I propose some changes: - Update binary name to `llama-cli` (for more details, see this PR: ggerganov/llama.cpp#7809 and this [homebrew formula](https://github.com/Homebrew/homebrew-core/blob/03cf5d39d8bf27dfabfc90d62c9a3fe19205dc2a/Formula/l/llama.cpp.rb)) - Add method to download llama.cpp via pre-built release - Split snippet into 3 sections: `title`, `setup` and `command` - Use `--conversation` mode to start llama.cpp in chat mode (chat template is now supported, ref: ggerganov/llama.cpp#8068) --- Proposal for the UI: (Note: Maybe the 3 sections title - setup - command can be more separated visually) 
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR brings the same logic of chat template from server to main (llama-cli).
Goals
llama_chat_apply_templatefunction--chat-templateargumentllama_chat_apply_templateand thus requires additional maintenance.How it works
llama_chat_apply_templatethat supportstd::string==> simplify the codellama_chat_format_single==> it evaluates the history twice, once with and once without the added message, then return the diffDemo
Fix #8053 #6391
Replace #6810