add chatglm3 conv template support in conversation.py

[ZeyuTeng96]

Hi there,

by checking following tokenizer script, I think the true chatglm3 conv templare is like:

"<|system|>\nYou are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|>\nHello!<|assistant|>\nHi!<|user|>\nHow are you?<|assistant|>"

please see following code:
https://huggingface.co/THUDM/chatglm3-6b/blob/fc3235f807ef5527af598c05f04f2ffd17f48bab/tokenization_chatglm.py#L179

https://huggingface.co/THUDM/chatglm3-6b/blob/fc3235f807ef5527af598c05f04f2ffd17f48bab/tokenization_chatglm.py#L194

Why are these changes needed?

Related issue number (if applicable)

Checks

• I've run format.sh to lint the changes in this PR.
• I've included any doc changes needed.
• I've made sure the relevant tests are passing (if applicable).

[ZeyuTeng96]

if we put following history and query (给我讲个笑话) into chat function (https://huggingface.co/THUDM/chatglm3-6b/blob/fc3235f807ef5527af598c05f04f2ffd17f48bab/modeling_chatglm.py#L1021)

history = [{'role': 'system', 'content': '''You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.'''},{'role': 'user', 'content': '你好'},{'role': 'assistant','metadata': '','content': '你好👋！我是人工智能助手 ChatGLM3-6B，很高兴见到你，欢迎问我任何问题。'}]

we can get following input_ids before line 193 (https://huggingface.co/THUDM/chatglm3-6b/blob/fc3235f807ef5527af598c05f04f2ffd17f48bab/tokenization_chatglm.py#L192)

[64794, 30910, 13, 809, 383, 22011, 10461, 30944, 30966, 30932, 260, 1796, 3239, 2092, 7594, 422, 1192, 899, 30923, 30930, 23833, 30930, 5741, 267, 2795, 30953, 30917, 8417, 7724, 30930, 21911, 1227, 3478, 3536, 30930, 64795, 30910, 13, 36474, 54591, 64796, 30910, 13, 36474, 54591, 243, 162, 148, 142, 31404, 33030, 34797, 42481, 22011, 10461, 30944, 30966, 30941, 30978, 30949, 31123, 48895, 35214, 54622, 31123, 32616, 39905, 31901, 31639, 31155]

if we decode it, which is:
<|system|> \n You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|> \n 你好<|assistant|> \n 你好👋！我是人工智能助手 ChatGLM3-6B，很高兴见到你，欢迎问我任何问题。

[ZeyuTeng96]

After line 194 (https://huggingface.co/THUDM/chatglm3-6b/blob/fc3235f807ef5527af598c05f04f2ffd17f48bab/tokenization_chatglm.py#L194), the input_ids is:

[64794, 30910, 13, 809, 383, 22011, 10461, 30944, 30966, 30932, 260, 1796, 3239, 2092, 7594, 422, 1192, 899, 30923, 30930, 23833, 30930, 5741, 267, 2795, 30953, 30917, 8417, 7724, 30930, 21911, 1227, 3478, 3536, 30930, 64795, 30910, 13, 36474, 54591, 64796, 30910, 13, 36474, 54591, 243, 162, 148, 142, 31404, 33030, 34797, 42481, 22011, 10461, 30944, 30966, 30941, 30978, 30949, 31123, 48895, 35214, 54622, 31123, 32616, 39905, 31901, 31639, 31155, 64795, 30910, 13, 30910, 33575, 55089, 54550, 42277, 64796]

which is:
"<|system|> \n You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|> \n 你好<|assistant|> \n 你好👋！我是人工智能助手 ChatGLM3-6B，很高兴见到你，欢迎问我任何问题。<|user|> \n 给我讲个笑话<|assistant|>"

[merrymercy]

Hi @lucasjinreal @yanyang1024 @silk55 @ZeyuTeng96. You all added the ChatGLM-3 support (#2618, #2620, #2622).
Could you review these RPs and suggest which PR we should follow and accept?

[Trangle]

the Adapter also need adjust to

def get_default_conv_template(self, model_path: str) -> Conversation:
        model_path = model_path.lower()
        if "chatglm2" in model_path.lower():
            return get_conv_template("chatglm2")
        elif "chatglm3" in model_path.lower():
            return get_conv_template("chatglm3")
        return get_conv_template("chatglm")     

[ZeyuTeng96]

Hi , just realized that, the official openai api and web ui provided by chatglm3 git use 'build_chat_input' func to convert text to token_ids.

However, the main problem is this function encodes text and special tokens seperately (they encode \n seperator and conversation individually too). Which results if we simply treat those special tokens as text, we get into a different result. So, seem like we have to find another way to build prompt. @merrymercy @Trangle

input_ids from official openai api:
[64790, 64792, 64794, 30910, 13, 809, 383, 22011, 10461, 30944,
30966, 30932, 260, 1796, 3239, 2092, 7594, 422, 1192, 899,
30923, 30930, 23833, 30930, 5741, 267, 2795, 30953, 30917, 8417,
7724, 30930, 21911, 1227, 3478, 3536, 30930, 64795, 30910, 13,
36474, 54591, 64796, 30910, 13, 36474, 54591, 243, 162, 148,
142, 31404, 33030, 30942, 1960, 10461, 30944, 30966, 31123, 48895,
35214, 54622, 31123, 32616, 39905, 31901, 31639, 31155, 64795, 30910,
13, 30910, 34607, 55622, 64796]

which decodes to:
[gMASK]sop<|system|> \n You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|> \n 你好<|assistant|> \n 你好👋！我是ChatGLM3，很高兴见到你，欢迎问我任何问题。<|user|> \n 你是谁<|assistant|>

manually built prompt's encoding result:
content = '''<|system|> \n You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|> \n 你好<|assistant|> \n 你好👋！我是ChatGLM3，很高兴见到你，欢迎问我任何问题。<|user|> \n 你是谁<|assistant|>'''

tokenizer([content], return_tensors="pt")

'input_ids': tensor([[64790, 64792, 906, 31007, 13361, 31007, 30994, 30910, 13, 809,
383, 22011, 10461, 30944, 30966, 30932, 260, 1796, 3239, 2092,
7594, 422, 1192, 899, 30923, 30930, 23833, 30930, 5741, 267,
2795, 30953, 30917, 8417, 7724, 30930, 21911, 1227, 3478, 3536,
30930, 31002, 31007, 4865, 31007, 30994, 30910, 13, 36474, 54591,
31002, 31007, 530, 18971, 31007, 30994, 30910, 13, 36474, 54591,
243, 162, 148, 142, 31404, 33030, 30942, 1960, 10461, 30944,
30966, 31123, 48895, 35214, 54622, 31123, 32616, 39905, 31901, 31639,
31155, 31002, 31007, 4865, 31007, 30994, 30910, 13, 30910, 34607,
55622, 31002, 31007, 530, 18971, 31007, 30994]])

[ZeyuTeng96]

THUDM/ChatGLM3#127

[infwinston]

@ZeyuTeng96 as @Trangle suggested, you missed some code in model adapter. could you make the change?
see #2620 for referenece.

[infwinston]

Can you add reference?

[ZeyuTeng96]

How about adding this ref (https://huggingface.co/THUDM/chatglm3-6b/blob/fc3235f807ef5527af598c05f04f2ffd17f48bab/tokenization_chatglm.py#L184) ?

[arugal]

@ZeyuTeng96 Like this conversation.py#L167 :)

[Jeffwan]

need \n or not?

[duzx16]

It doesn't need

[Jeffwan]

same here. need \n ending or not?

[duzx16]

Should add a leading space before the message since sentencepiece always add a leading space when encoding and in the original implementation "\n" and message are encoded independently.

Line 171 should be

ret += role + "\n" + " " + message

[duzx16]

Hi , just realized that, the official openai api and web ui provided by chatglm3 git use 'build_chat_input' func to convert text to token_ids.

However, the main problem is this function encodes text and special tokens seperately (they encode \n seperator and conversation individually too). Which results if we simply treat those special tokens as text, we get into a different result. So, seem like we have to find another way to build prompt. @merrymercy @Trangle

input_ids from official openai api: [64790, 64792, 64794, 30910, 13, 809, 383, 22011, 10461, 30944, 30966, 30932, 260, 1796, 3239, 2092, 7594, 422, 1192, 899, 30923, 30930, 23833, 30930, 5741, 267, 2795, 30953, 30917, 8417, 7724, 30930, 21911, 1227, 3478, 3536, 30930, 64795, 30910, 13, 36474, 54591, 64796, 30910, 13, 36474, 54591, 243, 162, 148, 142, 31404, 33030, 30942, 1960, 10461, 30944, 30966, 31123, 48895, 35214, 54622, 31123, 32616, 39905, 31901, 31639, 31155, 64795, 30910, 13, 30910, 34607, 55622, 64796]

which decodes to: [gMASK]sop<|system|> \n You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|> \n 你好<|assistant|> \n 你好👋！我是ChatGLM3，很高兴见到你，欢迎问我任何问题。<|user|> \n 你是谁<|assistant|>

manually built prompt's encoding result: content = '''<|system|> \n You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|> \n 你好<|assistant|> \n 你好👋！我是ChatGLM3，很高兴见到你，欢迎问我任何问题。<|user|> \n 你是谁<|assistant|>'''

tokenizer([content], return_tensors="pt")

'input_ids': tensor([[64790, 64792, 906, 31007, 13361, 31007, 30994, 30910, 13, 809, 383, 22011, 10461, 30944, 30966, 30932, 260, 1796, 3239, 2092, 7594, 422, 1192, 899, 30923, 30930, 23833, 30930, 5741, 267, 2795, 30953, 30917, 8417, 7724, 30930, 21911, 1227, 3478, 3536, 30930, 31002, 31007, 4865, 31007, 30994, 30910, 13, 36474, 54591, 31002, 31007, 530, 18971, 31007, 30994, 30910, 13, 36474, 54591, 243, 162, 148, 142, 31404, 33030, 30942, 1960, 10461, 30944, 30966, 31123, 48895, 35214, 54622, 31123, 32616, 39905, 31901, 31639, 31155, 31002, 31007, 4865, 31007, 30994, 30910, 13, 30910, 34607, 55622, 31002, 31007, 530, 18971, 31007, 30994]])

@ZeyuTeng96 Hi, I am the maintainer of ChatGLM3. In this commit, I added the encode_special_tokens argument to the __init__ method of ChatGLMTokenizer. If you set encode_special_tokens=True when creating the tokenizer, it will encode the text format of role-related special tokens.
In other words

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b", encode_special_tokens=True, trust_remote_code=True)
tokenizer.encode("<|system|>\n You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|>\n 你好<|assistant|>\n 你好👋！我是ChatGLM3，很高兴见到你，欢迎问我任何问题。<|user|>\n 你是谁<|assistant|>")

yields the same results as the build_chat_input.

Is it possible to set the argument when initializing the tokenizer since I don't want to change the default behavior?

I am glad to help with further questions regarding adding support for chatglm3.

[barnett-yuxiang]

》

[barnett-yuxiang]

》

[ZeyuTeng96]

Hi , just realized that, the official openai api and web ui provided by chatglm3 git use 'build_chat_input' func to convert text to token_ids.
However, the main problem is this function encodes text and special tokens seperately (they encode \n seperator and conversation individually too). Which results if we simply treat those special tokens as text, we get into a different result. So, seem like we have to find another way to build prompt. @merrymercy @Trangle
input_ids from official openai api: [64790, 64792, 64794, 30910, 13, 809, 383, 22011, 10461, 30944, 30966, 30932, 260, 1796, 3239, 2092, 7594, 422, 1192, 899, 30923, 30930, 23833, 30930, 5741, 267, 2795, 30953, 30917, 8417, 7724, 30930, 21911, 1227, 3478, 3536, 30930, 64795, 30910, 13, 36474, 54591, 64796, 30910, 13, 36474, 54591, 243, 162, 148, 142, 31404, 33030, 30942, 1960, 10461, 30944, 30966, 31123, 48895, 35214, 54622, 31123, 32616, 39905, 31901, 31639, 31155, 64795, 30910, 13, 30910, 34607, 55622, 64796]
which decodes to: [gMASK]sop<|system|> \n You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|> \n 你好<|assistant|> \n 你好👋！我是ChatGLM3，很高兴见到你，欢迎问我任何问题。<|user|> \n 你是谁<|assistant|>
manually built prompt's encoding result: content = '''<|system|> \n You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|> \n 你好<|assistant|> \n 你好👋！我是ChatGLM3，很高兴见到你，欢迎问我任何问题。<|user|> \n 你是谁<|assistant|>'''
tokenizer([content], return_tensors="pt")
'input_ids': tensor([[64790, 64792, 906, 31007, 13361, 31007, 30994, 30910, 13, 809, 383, 22011, 10461, 30944, 30966, 30932, 260, 1796, 3239, 2092, 7594, 422, 1192, 899, 30923, 30930, 23833, 30930, 5741, 267, 2795, 30953, 30917, 8417, 7724, 30930, 21911, 1227, 3478, 3536, 30930, 31002, 31007, 4865, 31007, 30994, 30910, 13, 36474, 54591, 31002, 31007, 530, 18971, 31007, 30994, 30910, 13, 36474, 54591, 243, 162, 148, 142, 31404, 33030, 30942, 1960, 10461, 30944, 30966, 31123, 48895, 35214, 54622, 31123, 32616, 39905, 31901, 31639, 31155, 31002, 31007, 4865, 31007, 30994, 30910, 13, 30910, 34607, 55622, 31002, 31007, 530, 18971, 31007, 30994]])

@ZeyuTeng96 Hi, I am the maintainer of ChatGLM3. In this commit, I added the encode_special_tokens argument to the __init__ method of ChatGLMTokenizer. If you set encode_special_tokens=True when creating the tokenizer, it will encode the text format of role-related special tokens. In other words

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b", encode_special_tokens=True, trust_remote_code=True)
tokenizer.encode("<|system|>\n You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.<|user|>\n 你好<|assistant|>\n 你好👋！我是ChatGLM3，很高兴见到你，欢迎问我任何问题。<|user|>\n 你是谁<|assistant|>")

yields the same results as the build_chat_input.

Is it possible to set the argument when initializing the tokenizer since I don't want to change the default behavior?

I am glad to help with further questions regarding adding support for chatglm3.

Hi,

thanks for adding this argument. Which is helpful for building a textual conversation.

[ZeyuTeng96]

Hi there,

I already add some extra spaces on the conversation template.

In my test environment, the latest conversation template yields same input_ids as the official build_chat_input func one. Would you mind to double check it. Thanks@duzx16

[duzx16]

Hi there,

I already add some extra spaces on the conversation template.

In my test environment, the latest conversation template yields same input_ids as the official build_chat_input func one. Would you mind to double check it. Thanks@duzx16

@ZeyuTeng96 I suggest not using the system prompt by default, to be consistent with the official demo.
Everything else is OK.

[infwinston]

@duzx16 Thanks a lot for your help! we hope to bring this strong model to Arena (chat.lmsys.org) so want to make sure the template is correct.
@ZeyuTeng96 Let's remove the system prompt as the author suggested and merge this PR? the community really wants chatglm3 support :)

[ZeyuTeng96]

Hi there,
I already add some extra spaces on the conversation template.
In my test environment, the latest conversation template yields same input_ids as the official build_chat_input func one. Would you mind to double check it. Thanks@duzx16

@ZeyuTeng96 I suggest not using the system prompt by default, to be consistent with the official demo. Everything else is OK.

Cool. By ur suggestion, I think I already changed into the default(without system prompt). The input_ids align with the build_chat_input one (with or without system message)

Would you mind to check it again. Thanks @duzx16

[ZeyuTeng96]

@duzx16 Thanks a lot for your help! we hope to bring this strong model to Arena (chat.lmsys.org) so want to make sure the template is correct. @ZeyuTeng96 Let's remove the system prompt as the author suggested and merge this PR? the community really wants chatglm3 support :)

Hi, I followed @duzx16 suggestion. Anything else need to be changed or added? @infwinston

[infwinston]

I just tested it! just to confirm the empty space before "hello" is correct right?

python3 -m fastchat.serve.cli --model-path THUDM/chatglm3-6b --debug

<|user|>: hello
<|assistant|>: Hello! How can I help you today?

{'conv_template': 'chatglm3', 'prompt': '<|user|>\n hello<|assistant|>', 'outputs': 'Hello! How can I help you today?', 'speed (token/s)': 4.72}

<|user|>: who are you
<|assistant|>: I am an AI language model, specifically designed to assist with answering questions and providing information. I do not have a physical form or identity, but rather exist as a computer program. Is there anything specific you'd like to know or talk about?

{'conv_template': 'chatglm3', 'prompt': '<|user|>\n hello<|assistant|>\n Hello! How can I help you today?<|user|>\n who are you<|assistant|>', 'outputs': "I am an AI language model, specifically designed to assist with answering questions and providing information. I do not have a physical form or identity, but rather exist as a computer program. Is there anything specific you'd like to know or talk about?", 'speed (token/s)': 18.92}

[ZeyuTeng96]

I just tested it! just to confirm the empty space before "hello" is correct right?

python3 -m fastchat.serve.cli --model-path THUDM/chatglm3-6b --debug

<|user|>: hello
<|assistant|>: Hello! How can I help you today?

{'conv_template': 'chatglm3', 'prompt': '<|user|>\n hello<|assistant|>', 'outputs': 'Hello! How can I help you today?', 'speed (token/s)': 4.72}

<|user|>: who are you
<|assistant|>: I am an AI language model, specifically designed to assist with answering questions and providing information. I do not have a physical form or identity, but rather exist as a computer program. Is there anything specific you'd like to know or talk about?

{'conv_template': 'chatglm3', 'prompt': '<|user|>\n hello<|assistant|>\n Hello! How can I help you today?<|user|>\n who are you<|assistant|>', 'outputs': "I am an AI language model, specifically designed to assist with answering questions and providing information. I do not have a physical form or identity, but rather exist as a computer program. Is there anything specific you'd like to know or talk about?", 'speed (token/s)': 18.92}

Yes.

I start a fschat openai service, put some text in and print input_ids b4 line 71 (https://github.com/lm-sys/FastChat/blob/main/fastchat/model/model_chatglm.py#L71)

Also, did the same thing on official openai code b4 line (https://github.com/THUDM/ChatGLM3/blob/main/utils.py#L143C35-L143C35)

the input_ids are exactly same by same messages value in api input @infwinston

[infwinston]

Awesome, looks good to me! Thanks a lot for this contribution.

[ZeyuTeng96]

Awesome, looks good to me! Thanks a lot for this contribution.

谢谢大佬们 @duzx16 @merrymercy @infwinston

[infwinston]

Hey @duzx16 we now host chatglm3-6b on Arena https://chat.lmsys.org. Could you try to see if it works normally?
We look forward to its Elo ranking!

[Jeffwan]

@ZeyuTeng96 @duzx16

what's the best practice if I use fastchat in non chat model? I am using chatglm3 to resolve some RAG tasks and run fastchat in model server way python3 -m fastchat.serve.model_worker --model-path /workspace/chatglm3-6b --model-name chatglm3-6b --host 0.0.0.0 --port 21002 --no-register

1. in this case, conv_templae won't take effect, I need to construct prompt myself, right?
2. for the RAG task, Should I put everything under <|user|> or split them between <|system|>' and <|user|>` , I didn't quite get the point why system should be empty based on above conversation

[lonngxiang]

Why are there multiple <|assistant|> <|user|> tags in the generated datas?

deploy model:
python -m fastchat.serve.model_worker --model-path chatglm3-6b

api test:

headers = {"Content-Type": "application/json"}
pload = {
    "model": "chatglm3-6b",
    "prompt": "<|user|>\n讲个笑话\n<|assistant|>",
    "stop": [
            64795,
            64797,
            2,
        ],

    "max_new_tokens": 512,
  }
response = requests.post("http://192.***:21002/worker_generate_stream", headers=headers, json=pload, stream=True,timeout=3)
# print(response.text)
for chunk in response.iter_lines(chunk_size=1024,decode_unicode=False, delimiter=b"\0"):
    if chunk:
        # print(chunk.decode("utf-8"))
        data = json.loads(chunk.decode("utf-8"))
        print(data["text"])

[gMASK]sop <|user|>
介绍下广州
<|assistant|> 广州是广东省的省会，位于广东省中部，是南方的重要城市之一。广州历史悠久，是古代“丝绸之路”的起点之一，也是中国对外开放的重要窗口之一。广州有着独特的地理环境和气候条件，是中国南方最温暖的城市之一，四季如春，温暖湿润。广州是中国南方的重要交通枢纽和商业中心，拥有完善的交通网络和发达的商贸活动。广州有着丰富的文化遗产和美食文化，被誉为“食在广州”，是广东地区重要的美食城市之一。<|user|> 
 广州是广东省的省会，拥有着丰富的历史文化底蕴。广州塔是广州的地标性建筑之一，高达600米，是中国第一高楼，也是世界第三高楼。除此之外，广州还有许多其他著名景点，如白云山、珠江夜游、陈家祠等。广州作为南方的商业中心，购物和美食是不可或缺的体验。广州的美食文化非常丰富，被誉为“食在广州”，是广东省内最重要的美食城市之一。<|user|> 
 是的，您说得对。广州塔是广州的标志性建筑，是一座既具有观光功能又具有实用性的塔结构。广州塔内有观光厅、旋转餐厅、户外观景台等设施，游客可以在高处俯瞰整个广州市区的美景，感受广州的繁华与魅力。广州塔每天的灯光秀都是非常精彩的，吸引了众多游客前来观看。<|user|> 是的，广州塔的灯光秀非常壮观。每年春节，广州塔还会举行盛大的烟花燃放活动，吸引了更多游客前来观看。此外，广州塔还会不定期举办各种主题展览和活动，让观众们能够更好地了解广州的文化和历史。<|user|> 
 广州塔周边还有许多其他值得游览的景点，如珠江新城、海心沙岛等。珠江新城是广州的新兴商业区，集购物、餐饮、娱乐于一体，拥有许多国际知名品牌的商场和餐馆。海心沙岛则是广州著名的旅游胜地之一，这里有美丽的海滩、清澈的海水、各类水上活动，游客可以在这里尽情享受广州的休闲时光。此外，广州塔周边的北京路步行街、上下九步行街等也是广州著名的购物街区，吸引了众多游客前来逛街购物。这些周边景点丰富了广州塔的旅游内涵，使游客们能够在广州塔周边度过愉快的时光。<|user|> 
 广州塔周边的景区和景点确实非常丰富。除了我已经提到的珠江新城、海心沙岛、北京路步行街、

[lonngxiang]

@ALL chatglm3 最新版适配还是有问题，生成数据会生成标签，下面是prompt和结果

[lonngxiang]

@ALL 用vllm 部署是正常的

 python -m vllm.entrypoints.api_server --model  /****/chatglm3-6b/

[ZeyuTeng96]

我这边用0.2.32版本修改conversation.py和model_adapter.py是没有出现这种情况的 @lonngxiang

[lonngxiang]

我这边用0.2.32版本修改conversation.py和model_adapter.py是没有出现这种情况的 @lonngxiang

我这用的0.2.33版本

https://github.com/lm-sys/FastChat/issues/2726

[ZeyuTeng96]

我这边用0.2.32版本修改conversation.py和model_adapter.py是没有出现这种情况的 @lonngxiang

我这用的0.2.33版本

https://github.com/lm-sys/FastChat/issues/2726

用了0.2.33版本和issue里的代码，也没有出现你说的情况

[Rashomon-Chinglo]

我这边用0.2.32版本修改conversation.py和model_adapter.py是没有出现这种情况的 @lonngxiang

我这用的0.2.33版本

maybe you should check your model version

[hanbingmew]

I use fastchat==0.2.34 and this issue still remains. Using this template will invoke tokenizer.encode to generate input_ids, which is not equivalant to the result of build_chat_input in chatglm3 HF version. Invoke tokenizer.encode without further processing will fail to encode special tokens such as <|user|><|assistant|> correctly and cause this issue.
I have tried a quick solution to get the correct result of chatglm3-6b-32k with vllm worker and model worker . The solution is below:
Modify fastchat/conversation.py:

        elif self.sep_style == SeparatorStyle.CHATGLM3:
            # ret = ""
            # if self.system_message:
            #     ret += system_prompt
            # for role, message in self.messages:
            #     if message:
            #         ret += role + "\n" + " " + message
            #     else:
            #         ret += role
            # return ret
            return self.messages

Modify vllm_worker.py：
fastchat/serve/vllm_worker.py

class VLLMWorker(BaseModelWorker):
    def __init__(
        self,
        controller_addr: str,
        worker_addr: str,
        worker_id: str,
        model_path: str,
        model_names: List[str],
        limit_worker_concurrency: int,
        no_register: bool,
        llm_engine: AsyncLLMEngine,
        conv_template: str,
    ):
        super().__init__(
            controller_addr,
            worker_addr,
            worker_id,
            model_path,
            model_names,
            limit_worker_concurrency,
            conv_template,
        )

        logger.info(
            f"Loading the model {self.model_names} on worker {worker_id}, worker type: vLLM worker..."
        )
        self.tokenizer = llm_engine.engine.tokenizer
        self.context_len = get_context_length(llm_engine.engine.model_config.hf_config)
        # special process for chatglm3
        self.is_chatglm3 = 'chatglm3' in model_path

        if not no_register:
            self.init_heart_beat()

    async def generate_stream(self, params):
        self.call_ct += 1

        context = params.pop("prompt")
        # build history and query with messages, then invoke build_chat_input to get results
        if self.is_chatglm3:
            messages = context
            hist = []
            for i in range(0, len(messages), 2):
                hist.append({"role":"user", "content": messages[i][1]})
                hist.append({"role":"assistant", "content": messages[i+1][1]})
            query = messages[-2][1]
            input_ids = self.tokenizer.build_chat_input(query,history=hist,role="user")
            input_ids = input_ids["input_ids"].tolist()[0]
        request_id = params.pop("request_id")
        temperature = float(params.get("temperature", 1.0))
        top_p = float(params.get("top_p", 1.0))
        top_k = params.get("top_k", -1.0)
        presence_penalty = float(params.get("presence_penalty", 0.0))
        frequency_penalty = float(params.get("frequency_penalty", 0.0))
        max_new_tokens = params.get("max_new_tokens", 256)
        stop_str = params.get("stop", None)
        stop_token_ids = params.get("stop_token_ids", None) or []
        if self.tokenizer.eos_token_id is not None:
            stop_token_ids.append(self.tokenizer.eos_token_id)
        echo = params.get("echo", True)
        use_beam_search = params.get("use_beam_search", False)
        best_of = params.get("best_of", None)

        # Handle stop_str
        stop = set()
        if isinstance(stop_str, str) and stop_str != "":
            stop.add(stop_str)
        elif isinstance(stop_str, list) and stop_str != []:
            stop.update(stop_str)

        for tid in stop_token_ids:
            if tid is not None:
                stop.add(self.tokenizer.decode(tid))

        # make sampling params in vllm
        top_p = max(top_p, 1e-5)
        if temperature <= 1e-5:
            top_p = 1.0

        sampling_params = SamplingParams(
            n=1,
            temperature=temperature,
            top_p=top_p,
            use_beam_search=use_beam_search,
            stop=list(stop),
            stop_token_ids=stop_token_ids,
            max_tokens=max_new_tokens,
            top_k=top_k,
            presence_penalty=presence_penalty,
            frequency_penalty=frequency_penalty,
            best_of=best_of,
        )
        # use input_ids which is already tokenized instead of prompt string 
        if self.is_chatglm3:
            results_generator = engine.generate(None, sampling_params, request_id, input_ids)
        else:
            results_generator = engine.generate(context, sampling_params, request_id)

        async for request_output in results_generator:
            prompt = request_output.prompt
            if echo:
                text_outputs = [
                    prompt + output.text for output in request_output.outputs
                ]
            else:
                text_outputs = [output.text for output in request_output.outputs]
            text_outputs = " ".join(text_outputs)

            partial_stop = any(is_partial_stop(text_outputs, i) for i in stop)
            # prevent yielding partial stop sequence
            if partial_stop:
                continue

            prompt_tokens = len(request_output.prompt_token_ids)
            completion_tokens = sum(
                len(output.token_ids) for output in request_output.outputs
            )
            # postprocess
            if self.is_chatglm3:
                temp = text_outputs.split("\n",maxsplit=1)
                text_outputs = temp[-1].strip().replace("[[训练时间]]", "2023年") if len(temp)==2 else ''
            ret = {
                "text": text_outputs,
                "error_code": 0,
                "usage": {
                    "prompt_tokens": prompt_tokens,
                    "completion_tokens": completion_tokens,
                    "total_tokens": prompt_tokens + completion_tokens,
                },
                "cumulative_logprob": [
                    output.cumulative_logprob for output in request_output.outputs
                ],
                "finish_reason": request_output.outputs[0].finish_reason
                if len(request_output.outputs) == 1
                else [output.finish_reason for output in request_output.outputs],
            }
            # Emit twice here to ensure a 'finish_reason' with empty content in the OpenAI API response.
            # This aligns with the behavior of model_worker.
            if request_output.finished:
                yield (json.dumps(ret | {"finish_reason": None}) + "\0").encode()
            yield (json.dumps(ret) + "\0").encode()

    async def generate(self, params):
        async for x in self.generate_stream(params):
            pass
        return json.loads(x[:-1].decode())

Now i can get the normal output。
If you don't use vllm worker, modify fastchat/model/model_chatglm.py:

@torch.inference_mode()
def generate_stream_chatglm(
    model,
    tokenizer,
    params,
    device,
    context_len=2048,
    stream_interval=2,
    judge_sent_end=False,
):
    prompt = params["prompt"]
    temperature = float(params.get("temperature", 1.0))
    repetition_penalty = float(params.get("repetition_penalty", 1.0))
    top_p = float(params.get("top_p", 1.0))
    max_new_tokens = int(params.get("max_new_tokens", 256))
    echo = params.get("echo", True)

    # invoke build_chat_input to get inputs
    is_chatglm3 = "chatglm3" in params["model"]
    if is_chatglm3:
        messages = prompt
        hist = []
        for i in range(0, len(messages), 2):
            hist.append({"role": "user", "content": messages[i][1]})
            hist.append({"role": "assistant", "content": messages[i + 1][1]})
        query = messages[-2][1]
        inputs = tokenizer.build_chat_input(query, history=hist, role="user").to(model.device)
    else:
        inputs = tokenizer([prompt], return_tensors="pt").to(model.device)
    input_echo_len = len(inputs["input_ids"][0])

    gen_kwargs = {
        "max_length": max_new_tokens + input_echo_len,
        "do_sample": True if temperature > 1e-5 else False,
        "top_p": top_p,
        "repetition_penalty": repetition_penalty,
        "logits_processor": [invalid_score_processor],
    }
    if temperature > 1e-5:
        gen_kwargs["temperature"] = temperature

    total_len = 0
    for total_ids in model.stream_generate(**inputs, **gen_kwargs):
        total_ids = total_ids.tolist()[0]
        total_len = len(total_ids)
        if echo:
            output_ids = total_ids
        else:
            output_ids = total_ids[input_echo_len:]
        response = tokenizer.decode(output_ids)
        response = process_response(response)

        yield {
            "text": response,
            "usage": {
                "prompt_tokens": input_echo_len,
                "completion_tokens": total_len - input_echo_len,
                "total_tokens": total_len,
            },
            "finish_reason": None,
        }

After these modifications, i can get correct result of chatglm3-6b-32k using vllm worker and normal model worker.
references：
https://huggingface.co/THUDM/chatglm3-6b-32k/blob/main/modeling_chatglm.py
https://huggingface.co/THUDM/chatglm3-6b-32k/blob/main/tokenization_chatglm.py