[Bug]: Gemma model fails with GPTQ marlin

[arunpatala]

🐛 Describe the bug

Using docker and gemma finetuned model

--model /data/merged_model_GPTQ --max-model-len 8192 --max-num-seqs 1024 --served-model-name model --quantization gptq_marlin

fails with

RuntimeError: Some weights are not initialized from checkpoints: {'model.layers.3.mlp.gate_up_proj.g_idx_sort_indices', 'model.layers.8.self_attn.qkv_proj.g_idx_sort_indices', 'model.layers.9.mlp.gate_up_proj.g_idx_sort_indices .....

The same works with --quantization gptq

[robertgshaw2-neuralmagic]

🐛 Describe the bug

Using docker and gemma finetuned model

--model /data/merged_model_GPTQ --max-model-len 8192 --max-num-seqs 1024 --served-model-name model --quantization gptq_marlin

fails with

RuntimeError: Some weights are not initialized from checkpoints: {'model.layers.3.mlp.gate_up_proj.g_idx_sort_indices', 'model.layers.8.self_attn.qkv_proj.g_idx_sort_indices', 'model.layers.9.mlp.gate_up_proj.g_idx_sort_indices .....

The same works with --quantization gptq

@alexm-neuralmagic

@arunpatala Can you share the model checkpoint so we can take a look?

[arunpatala]

volume=$HF_HOME
docker run --runtime nvidia --gpus all \
    -v $volume:/root/.cache/huggingface \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model TechxGenus/gemma-1.1-2b-it-GPTQ \
    --max-model-len 8192 \
    --max-num-seqs 32 \
    --quantization gptq_marlin

You can check with this public model. The engine throws error when using either gptq_marlin or marlin as quantization.
RuntimeError: Some weights are not initialized from checkpoints: {'model.layers.6.self_attn.qkv_proj.g_idx_sort_indices',

The same model works with gptq as quantization.

volume=$HF_HOME
docker run --runtime nvidia --gpus all \
    -v $volume:/root/.cache/huggingface \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model TechxGenus/gemma-1.1-2b-it-GPTQ \
    --max-model-len 8192 \
    --max-num-seqs 32 \
    --quantization gptq

[robertgshaw2-neuralmagic]

Thanks

@alexm-neuralmagic can you take a look?

[alexm-neuralmagic]

@arunpatala Here is the fix #5108 (will land soon)

[arunpatala]

thanks a lot.

[alexm-neuralmagic]

no problem