refact : fix convert script + zero out KV cache to avoid nans

[ggerganov]

ref: #3329 (comment)

• Copied tokenization from convert-starcoder-hf-to-gguf.py
• ALiBi is prone to random KV cache data, so we have to zero out the cache at the start. Since llama : custom attention mask + parallel decoding + no context swaps #3228, we can access uninitialized KV cache data due to:
  

  llama.cpp/llama.cpp

  Line 5024 in bdbe117

  kv_self.n = std::min((int32_t) cparams.n_ctx, std::max(32, llama_kv_cache_cell_max(kv_self)));

  
  If this data happens to contain nan, then the generation fails

Question: should we first mask the KV tensor and then apply ALiBi?

llama.cpp/llama.cpp

Lines 3763 to 3771 in bdbe117

	// KQ_masked = mask_past(KQ_scaled)
	struct ggml_tensor * KQ_scaled_alibi = ggml_alibi(ctx0, KQ_scaled, /*n_past*/ 0, n_head, 8);
	ggml_set_name(KQ_scaled_alibi, "KQ_scaled_alibi");
	struct ggml_tensor * KQ_masked = ggml_add(ctx0, KQ_scaled_alibi, KQ_mask);
	offload_func_kq(KQ_masked);
	ggml_set_name(KQ_masked, "KQ_masked");

If that were the case, then the above KV cache initialization wouldn't be needed since any uninitialized values will be masked with -INF

[slaren]

If that were the case, then the above KV cache initialization wouldn't be needed since any uninitialized values will be masked with -INF

But nan - INF is still nan, so I don't think that this would work for removing nans before alibi.