Finetuning with LLaMA-Efficient-Tuning and deploying with fastchat， but get poor result

[myj951]

I have fine-tuned the Baichuan2 model using another framework and deployed it using fastchat. However, I encountered the following issue: the accuracy on fastchat decreased. Here is the deployment code I used:

python3 -m fastchat.serve.cli --model-path /backup/baichuan/912/baichuan2-fintuned-50b
I would like to know which parameters I should modify to solve this problem and I would appreciate guidance on parameter selection if possible. If it is not possible to solve this issue, does it mean that I can only achieve the same results by fine-tuning and deploying within the fastchat framework?

The fine-tuning project I used is available at:
https://github.com/hiyouga/LLaMA-Efficient-Tuning/blob/main/README_zh.md
By using the local deployment demo provided by this project, I obtained consistent results (overfitting with 99%+ accuracy on the training set, and good generated outputs when deploying the model).

[ye7love7]

I tried to deploy baichuan2-13b-chat by fastchat, but the performance is very different from that of normal deployment. I don't know why the performance is lost,is this same to you?

[myj951]

I tried to deploy baichuan2-13b-chat by fastchat, but the performance is very different from that of normal deployment. I don't know why the performance is lost,is this same to you?

Yes，that's my question. I tried both baichuan and baichuan2 and this problem occurred. I wonder if this is due to the difference in default parameters between the training project and the deployment project.

[Betai18n]

Because baichuan2 use new template with <reserved_106> and <reserved_107>, and I find fastchat add support for baichuan2 models just now. #2408

[ye7love7]

thanks！

…

---Original--- From: ***@***.***&gt; Date: Wed, Sep 13, 2023 17:21 PM To: ***@***.***&gt;; Cc: ***@***.******@***.***&gt;; Subject: Re: [lm-sys/FastChat] Finetuning with LLaMA-Efficient-Tuning and deploying with fastchat， but get poor result (Issue #2407) Because baichuan2 use new template with <reserved_106&gt; and <reserved_107&gt;, and I find fastchat add support for baichuan2 models just now. #2408 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***&gt;