How to load the original model LM head in the new version of Adapters?

[Guitaricet]

Details

Hello!

We are working with LLaMA which is not supported in the old version of Adapter-Transformers. So we decided to use the beta (Adapters branch). The issue is, it's unclear how to load the llama language modeling head. By default, the model is loaded without any head, which is a problem. This kind of approach made a lot of sense in the BERT era, but it's unclear if it's an intuitive solution now when people want to fine-tune language models with their original head on.

Here's what we tried:

from adapters import LlamaAdapterModel
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
model = LlamaAdapterModel.from_pretrained("meta-llama/Llama-2-7b-hf")

model.add_causal_lm_head("lm_head")
adapters.init(model)
out = model.generate(tokenizer("The cat is", return_tensors="pt")["input_ids"], max_length=20)
tokenizer.decode(out[0])
# output: '<s> The cat isРСРoulevalu str siècleTest materadejkradeavenradebosebose étudesvä'

so, it seems like this way the added head is randomly initialized.

Could you help us to get the original llama head weights?

[hSterz]

Hello @Guitaricet, yes you are correct the add_causal_lm_head adds a randomly initialized language modelling head. What you can do here is you can use the original transformers class and initialize it afterwards:

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
adapters.init(model)

This should solve the problem (let me know if it does not).

[Guitaricet]

Thank you, everything works now!