position_ids is now required by FalconRotaryEmbedding forward method

[iosonopersia]

Hi everyone, I was experimenting with the "tiiuae/falcon-7b-instruct" model and I found an issue.

My setup:

optimum==1.13.2
torch==2.0.1+cu118
transformers==4.34.0

The problem is that, since this PR (huggingface/transformers#26137) was merged (i.e. v4.34.0 onwards), Falcon models require the "position_ids" argument when calling the forward() pass of the FalconRotaryEmbedding module (see here).

I saw that thanks to PR #1421, the falcon_forward() method now accepts any additional positional argument, most notably including the "position_ids" one (which is in turn required by the self.maybe_rotary() method, as it appears to call FalconRotaryEmbedding.forward()):

optimum/optimum/bettertransformer/models/attention.py

Lines 939 to 940 in 8c296d3

	past_kv_length = 0 if layer_past is None else layer_past[0].shape[1]
	query_layer, key_layer = self.maybe_rotary(query_layer, key_layer, past_kv_length)

My workaround fix is the following:

 if "position_ids" in kwargs:
        past_kv_length = 0 if layer_past is None else layer_past[0].shape[1]
        query_layer, key_layer = self.maybe_rotary(query_layer, key_layer, past_kv_length, kwargs["position_ids"])

I don't know if this is the best way to deal with the issue, as it doesn't handle the case in which "position_ids" is not provided (I don't even know if there are cases in which it could happen). Moreover, I didn't check whether the same applies for other LLM models too.

[fxmarty]

Hi @iosonopersia, thank you for the report! This is fixed in #1431. So as to keep the maintenance burden low, BetterTransformer for Falcon will be supported only for transformers 4.34 with a meaningful message in case a lower version is installed:

ImportError: FalconAttentionLayerBetterTransformer requires the transformers>=4.34 library but it was not found in your environment. You can install it with pip: pip install -U transformers. Please note that you may need to restart your runtime after installation.