[compiler] support RoPE Op

[ys44kim]

What

Lets' support RoPE (Rotary Position Embedding) and fuse

The image below is part of the token generation model.
This graph is fused with rope op.

Conditions

Input

• (1, attention head num, 1, head_dim)
• sin_table : (1,1,1,head_dim), the value determined depending on what position the token is in in the sequence.
• cos_table : (1,1,1,head_dim), the value determined depending on what position the token is in in the sequence.

Task

• luci.lang
• circle_schema
• luci.import
• res/TensorFlowLiteRecipes, res/CircleRecipes
• luci.service
• luci.pass
• luci.partition
• luci.logex
• luci.export
• luci_interpreter
• circle.dump
• circle2circle
• luci.tests
• luci-pass-value-test
• circle2circle-dredd-recipe-test

[jinevening]

FYI, There are two different types of RoPE implementation.

1. GPT-NeoX style
2. GPT-J style
   (See [LLaMA] Rotary positional embedding differs with official implementation huggingface/transformers#25199 for more details)

The RoPE you want to support is NeoX style. It would be better to mention that somewhere (maybe luci.lang).

[seanshpark]

two different types of RoPE implementation.

Do they have different I/O, attribute configuration?
or just has different decomposed graph?

[jinevening]

I think they may have the same I/O, but have different op patterns.

[glistening]

@jinevening and @seanshpark

I talked with @hseok-oh in runtime perspective and circle schema maintenance.

• I think two different styles of neo-x and gpt could be in a circle op by adding mode parameter as llama.cpp does.
• We can go with only neo-x for a while.
• Historically, we add new op in backward from 255. I would like to use 1 op if possible to allocate two op slots for rope.

Consequently, if compiler is okay, we may go with rope op (implicitly neo-x by default).
Then, we can try to add gpt style when gpt becomes necessary in the same op.
If it does not go as we expected, we may add a second new op like rope_gpt_style.

[ys44kim]

Draft: #13978

[jinevening]

I'm ok with #13972 (comment) if @seanshpark approves :)

[ys44kim]

I talked to seanshpark offline,
i will add mode to schema and luci as below.
8dbd1df

enum RoPEMode : byte {
  NEOX,
  GPT,
}
table RoPEOptions {
  mode: RoPEMode;
}

enum class RoPEMode
{
  UNDEFINED, // This is not defined by Circle. This was added to prevent programming error.

  NEOX,
  GPT,
};

[glistening]

@ys44kim Could you please use GPTJ instead of GPT? GPT seems too broad ¹.

Footnotes

1. https://github.com/vllm-project/vllm/issues/747
   https://github.com/huggingface/transformers/issues/25199
   https://velog.io/@nawnoes/ALiBi-Postion-Embedding ↩

[ys44kim]

@ys44kim Could you please use GPTJ instead of GPT? GPT seems too broad 1.

@glistening
I was thinking about what to do with the enum name. so, I wanted to hear your opinion on what to do.
i think NEOX and GPTJ, NEOX has no prefix and GPTJ has prefix, so I was worried about how to deal with GPT.
so, GPT_NEOX, GPT_J?

[glistening]

@ys44kim GPT_NEOX and GPT_J is also okay for me.