T5 AdapterDrop Prefix-Tuning Bug

[FahadEbrahim]

Hi,

I'm trying to use train an AdapterDrop with T5. It's working fine for bottleneck, LoRA and IA3. But it does not work on Prefix Tuning (Therefore MAM and UniPELT).

Following is a notebook:
https://gist.github.com/FahadEbrahim/66686814f02978da9d4376470356647d

The error is:
RuntimeError: The size of tensor a (90) must match the size of tensor b (80) at non-singleton dimension 3.

[TimoImhof]

Hi @FahadEbrahim,

The bug happens only for Prefix Tuning and not the other adapter methods because Prefix Tuning changes the input dimensions when adding the prefixes. When now dropping adapter layers during training with AdapterDrop and Prefix Tuning, the individual transformer layers of T5 can have different dimensions. This leads to the runtime error because the positional encoding in T5 is always forwarded to the next layer, assuming the dimensions will never change.

Thanks for bringing this up! With the new PR, this problem should be solved; your script is running fine with the fix on my machine.

[FahadEbrahim]

@TimoImhof Thank you for your quick response and feedback. I tested the new PR branch and it's working perfectly.

With my appreciation,
Fahad.