[Bugfix] Fix prefix strings for quantized VLMs

[mgoin]

The prefix strings were not being constructed correctly for many VLMs, particularly due to the use of sub-models. This was causing quant_config to be enabled for modules that are ignored within the quantization_config of the model, causing failures during weight loading.

It's important to note that the "XForCausalLM" classes can be used as sub-models in VLMs (Gemma and Llama are easy examples), so we need to have the ability to optionally specify the prefix to those classes. For example in Llama you could have a module name model.layers.0.mlp.down_proj, but when it is used within a VLM that module name should be language_model.model.layers.0.mlp.down_proj
I propose a maybe_prefix(prefix, name) function that handles this logic.

I think in the future we should consider making prefix a required argument in many cases to make issues less likely, rather than a default empty string.

Tested models:

• nm-testing/MiniCPM-V-2_6-FP8-dynamic
• nm-testing/paligemma-3b-mix-224-FP8-dynamic
• neuralmagic/Llama-3.2-11B-Vision-Instruct-FP8-dynamic
• nm-testing/Qwen2-VL-7B-Instruct-FP8-dynamic
• nm-testing/pixtral-12b-FP8-dynamic
• nm-testing/Phi-3.5-vision-instruct-FP8-dynamic
• nm-testing/llava-onevision-qwen2-7b-ov-hf-FP8-dynamic

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

Thanks for taking the time and effort for adding this! I think this should solve most of the quantization-related GH issues?

• [Feature Request]: Support INT4 for MiniCPM-Llama3-V-2_5 #6932
• [Usage]: How to use FP8 or other quantization algorithms for Minicpmv2_6 #7724
• [New Model]: MiniCPM-V-2_6-int4 #7727
• [New Model]: We can able to run phi-3.5 vision instruct model but wanted to run in int4 quantization  #8463
• [Feature]: Quantization support for LLaVA OneVision  #9324

(Phi3V and Llava-OneVision haven't been tested though)

[mgoin]

@DarkLight1337 A few of the issues you linked (and shared earlier with me) are specific to bitsandbytes support, so will require more work but I want to tackle those in future PRs since BNB is special

[DarkLight1337]

LGTM then. See if @Isotr0py is ok with this approach as well.

[Isotr0py]

LGTM!

BTW, if we want to broadcast the BNB support to most of VLMs, we might need to implement quantized multi_modal_proj (if they have) as well. Because user will possibly create a fully quantized (even the lm_head and multi_modal_proj) BNB model if they refer to the guidance in transformers...