Can not infer quantized model, but fp32 works well.

[znsoftm]

run main.exe with a q4_0 quantized model.

for q4_0, it complains assert in ggml.c

in function static void ggml_compute_forward_soft_max_f32(

line 9342: assert(sum > 0.0); sum is -nan (ind)

[znsoftm]

overflowed?

[znsoftm]

But q4_1 works well.

[skeskinen]

nan results are typically a sign of some float accuracy weirdness. Do you have a very small model? I think the quantization is less accurate the smaller your model is.

[znsoftm]

get a quntized model from this model: multi-qa-MiniLM-L6-cos-v1 on hugging face.

[znsoftm]

I modify the code to adapt to BertCode with the latest ggml, it works fine. Maybe it can be solve by upgrading GGML?

[znsoftm]

nan results are typically a sign of some float accuracy weirdness. Do you have a very small model? I think the quantization is less accurate the smaller your model is.
ggml-model-q4_0.zip

[appvoid]

run main.exe with a q4_0 quantized model.

for q4_0, it complains assert in ggml.c

in function static void ggml_compute_forward_soft_max_f32(

line 9342: assert(sum > 0.0); sum is -nan (ind)

Can you please tell me how did you manage to make it work on Windows???

[znsoftm]

I have pulled a request and the repo owner has merged it. Git pull to get a new version, it works on Windows.

[appvoid]

Thanks, it's working now!