HELM evaluation framework tries to generate with temperature=0 which leads to inf tensors

[aniketmaurya]

HELM evaluation framework tries to generate with temperature=0 which leads to inf tensors error.

I don't have much idea how LLM will react to adding 1e-9 to the temperature value but this would avoid the error here.

        # forward
        logits = model(x, max_seq_length, input_pos)
        logits = logits[0, -1] / (temperature + 1e-9)

[carmocca]

Do you know why is that? It's very strange. Do you have any links?

[aniketmaurya]

I don't have any link, unfortunately! I tried printing the input request where the temperature was set to 0.

[Andrei-Aksionov]

Weird assumption: this framework relies on this additive in the denominator part and thus wants to use the lowest possible temperature so during sampling the logits with the highest probability will be outputted.
Though not sure how useful this information is 🙃.

[carmocca]

There's precedent for this: https://github.com/stanford-crfm/helm/blob/de02a1e05fa438d718f9a22dc5cff40f33f8ee41/src/helm/proxy/clients/huggingface_client.py#L188

[aniketmaurya]

thanks for finding this @carmocca! It seems that neurips client doesn't have this condition to the raw request - https://github.com/stanford-crfm/helm/pull/1693/files#diff-a9de0c80d20516c09d5131d26c59dfdab923a2c7cfd2e09b8814e9bc4bd7521eR47

[aniketmaurya]

stanford-crfm/helm#1693 will close this issue