Applying parallel attn with ff to existing pretrained model?

[huu4ontocord]

Hi - awesome work! I am trying to understand ? I couldn't find a paper - only a reference to https://github.com/kingoflolz/mesh-transformer-jax. Is this right? Am I understanding that it is bascially applying multiple operations of for qkv and ff at once? Is it possible to use this trick to modify an existing pretrained model?

flamingo-pytorch/flamingo_pytorch/flamingo_palm.py

Line 90 in 749f824

# parallel attention and feedforward with residual

Many thanks in advance!

Huu

[lucidrains]

@ontocord yup that's correct, it was invented by Ben Wang for GPT-J, then subsequently adopted by PaLM