[BUG]: Object Reference Error in ApplyPenalty when setting nl_logit

[GalactixGod]

Description

This error occurs when using Kernel Memory and can be reproduced when running the
Kernel Memory: Save and Load example and appears to be reproducible (at least by me) regardless of which model I try.

I have tried on two different systems with two distinct hardware configurations and the error persists between both setups. One is CPU bound and the other has a Nvidia 4080.

Reproduction Steps

Execute the Example Program. Allow the program to execute. When the question "What formats does KM support" is asked. The application will fail.

Please note:
The data from the files appears to be successfully tokenized and saved in the storage folder.

Once a question is and the ApplyPenalty method executes there is a line:
var nl_logit = logits[(int?)nl_token ?? 0];

This code consistently generates an Object Reference Not Set To An Instance of An Object. The variable appears to be set a few lines earlier:
var logits = NativeHandle.GetLogitsIth(logits_i);

I can't seem to debug this as any attempt to see what is contained within logits results in a Visual Studio error indicating an error while evaluating the SPAN.

Environment & Configuration

• Operating system: Windows 11
• .NET runtime version: .NET 8
• LLamaSharp version: latest for GIT - 0.14.X
• CUDA version (if you are using cuda backend): Compiling llamasharp directly and running examples
• CPU & GPU device: Error occurs on various PCs of different configurations.

Known Workarounds

No response

[axeok]

+1

[martindevans]

If that's throwing it must mean the Span<float> logits is pointing to an invalid memory location (that would also be why you can't inspect it in the debugger, trying to do so will attempt to read the invalid memory).

To help debug this, what is the value of logits_i when it happens?

[GalactixGod]

The value of logits_i at the time of the error is 299.

here's a youtube link so you can see what I'm seeing:
https://youtu.be/sTlvAdf2vwA