Skip to content

Commit

Permalink
Smol Fix (huggingface#11)
Browse files Browse the repository at this point in the history
  • Loading branch information
saurabhdash2512 authored Mar 13, 2024
1 parent 82a0f3b commit ef6ed3d
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions src/transformers/models/cohere/tokenization_cohere_fast.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,6 @@ def default_chat_template(self):
<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>{{ I am doing well! }}<|END_OF_TURN_TOKEN|>
Use add_generation_prompt to add a prompt for the model to generate a response:
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01", trust_remote_code=True)
>>> messages = [{"role": "user", "content": "Hello, how are you?"}]
Expand Down Expand Up @@ -742,4 +741,4 @@ def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
if token_ids_1 is not None:
output = output + bos_token_id + token_ids_1 + eos_token_id

return output
return output

0 comments on commit ef6ed3d

Please sign in to comment.