How to Enable Fully Greedy Decoding Sample Parameters?

[PenutChen]

Hi,

I'm planning to employ in-context learning in my project and have chosen to use greedy decoding. Unlike in HuggingFace, it seems there is no do_sample=False parameter. Based on my research, I've established the following parameters for greedy decoding:

param = {
    "n_predict": 256,
    "stop": ["\n\n"],
    "prompt": prompt,
    "temperature": 0.0,
    "top_k": 0,
    "top_p": 0.0,
    "repeat_last_n": 0,
    "repeat_penalty": 1.0,
    "penalize_nl": False,
    "tfs_z": 1.0,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0,
    "mirostat": 0
}

I've chosen these parameters based on the server documentation since I'm using the server to serve as an LLM backend. Can anyone provide feedback on this? Specifically, I'm wondering if I've missed something or if there are better values for certain parameters given my intended use-case.

Thank you in advance!

[KerfuffleV2]

At least in the main example, setting temperature to a negative value enables greedy sampling. edit: I checked, seems the same in examples/server.

[PenutChen]

My apologies for overlooking the details about the temperature parameter. I had subjectively assumed that setting the temperature to 0 would simply disable temperature sampling, while other methods like top_k and top_p would continue to function as normal. To my understanding, these sampling methods should operate independently, correct? Thank you once again for your response!

[KerfuffleV2]

No problem. I'm not sure I 100% understand your question. For greedy sampling, stuff like top_k won't apply. The code looks like this:

llama.cpp/common/common.cpp

Lines 927 to 946 in 2ba85c8

	if (temp <= 0) {
	// Greedy sampling
	id = llama_sample_token_greedy(ctx, &cur_p);
	} else {
	if (mirostat == 1) {
	static float mirostat_mu = 2.0f * mirostat_tau;
	const int mirostat_m = 100;
	llama_sample_temperature(ctx, &cur_p, temp);
	id = llama_sample_token_mirostat(ctx, &cur_p, mirostat_tau, mirostat_eta, mirostat_m, &mirostat_mu);
	} else if (mirostat == 2) {
	static float mirostat_mu = 2.0f * mirostat_tau;
	llama_sample_temperature(ctx, &cur_p, temp);
	id = llama_sample_token_mirostat_v2(ctx, &cur_p, mirostat_tau, mirostat_eta, &mirostat_mu);
	} else {
	// Temperature sampling
	llama_sample_top_k (ctx, &cur_p, top_k, 1);
	llama_sample_tail_free (ctx, &cur_p, tfs_z, 1);
	llama_sample_typical (ctx, &cur_p, typical_p, 1);
	llama_sample_top_p (ctx, &cur_p, top_p, 1);
	llama_sample_temperature(ctx, &cur_p, temp);

So if temperature < 0 then you only get greedy sampling. If mirostat 1 or 2 is enabled, you only get temperature + mirostat, otherwise the main set of samplers like top-k, tail free, etc will apply. (Not included in the code fragment I linked is the penalty stuff like repeat penalty which applies in all cases when enabled.)

The way the temperature sampler works in llama.cpp doesn't actually select a token or anything, it just divides the logits by the temperature:

llama.cpp/llama.cpp

Lines 4065 to 4070 in 2ba85c8

	void llama_sample_temperature(struct llama_context * ctx, llama_token_data_array * candidates_p, float temp) {
	const int64_t t_start_sample_us = ggml_time_us();
	for (size_t i = 0; i < candidates_p->size; ++i) {
	candidates_p->data[i].logit /= temp;
	}

[PenutChen]

You've completely addressed my blind spots concerning the sampling methods and resolved my questions. I now understand that the temperature parameter is actually a kind of logits augmentation. Thank you for your thorough explanation!

[jackfsuia]

hi. can you ask questions continued on what you've discussed? I use
llama-server.exe -m "D:\my repo\qwen2\ggml-model-bf16.gguf --temp 0.0
to be my server. and Use

import openai
client = openai.OpenAI(
    base_url="http://localhost:8080/v1", 
    api_key = "sk-no-key-required"
)
completion = client.chat.completions.create(
    model="qwen",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "write a letter about love to a cat"}
    ],
)
print(completion.choices[0].message.content)

to be my request. but every time the response turn out different, which might suggest that the server was not doing greedy decoding. What did i do wrong? Thank you~

[jackfsuia]

Thank you for your help! I tried this, but once i passed temperature=0.0 in there, the server seemed to be stuck somewhere forever, and nothing happen, and i don't know why.

[PenutChen]

Maybe you can try using streaming output to see what happens:

completion = client.chat.completions.create(
    model="qwen",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "write a letter about love to a cat"}
    ],
    temperature=0.0,
    stream=True,
)

for chunk in completion:
    try:
        chunk_message = chunk.choices[0].delta.content
        print(end=chunk_message)
    except Exception as e:
        print(e)

[jackfsuia]

Thank you for your brilliant solution, it turns out the server was producing the same sentences non-stop. The problem is clear now. Can I ask one more question? When the llama.cpp runs the non-quantized .gguf model file, is it supposed to produce the same results as the original huggingface model for the same input? But I haven't succeed at making it do so. Is it possible to do so? Thank you again!

[PenutChen]

This is pretty difficult to align the responses of these backends. In my experience, not only does the temperature need to be set to 0.0, but also frequency_penalty, presence_penalty, or repeat-penalty (if they exist) need to be set properly. By the way, the most greedy decode of llama.cpp is example/simple.

Remember to check the generation config of HF Transformers. They also need to be set to greedy decode.

[jackfsuia]

You are right, it is difficult. I try llama-simple on the model QWEN2-0.5B, both f16.gguf and bf16.gguf failed to be aligned with hf. I give up. Thank you again.

[Zuo-Peng]

This means that if I want to use greedy decoding, I just need to set temp=0.

[ggerganov]

Setting temp = 0 will no longer be equivalent to greedy decoding (see #9897). To enable it, configure a single top_k sampler and set k = 1. For example, with llama-cli this can be done with the following CLI args:

--sampling-seq k --top-k 1