Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

model: support arch DbrxForCausalLM #6515

Merged
merged 81 commits into from
Apr 13, 2024
Merged

model: support arch DbrxForCausalLM #6515

merged 81 commits into from
Apr 13, 2024

Commits on Apr 6, 2024

  1. model: dbrx convert to gguf 

    #6344
    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    1d8de31 View commit details
    Browse the repository at this point in the history

  2. llama: support dbrx 

    #6344
    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    ed582c1 View commit details
    Browse the repository at this point in the history

  3. gguf-py: remove wrong clip -> clamp

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    3e3d2d1 View commit details
    Browse the repository at this point in the history

  4. model: dbrx, trust remote code

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    3937100 View commit details
    Browse the repository at this point in the history

  5. llama: add label for model 132B

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    c0beb3c View commit details
    Browse the repository at this point in the history

  6. model: dbrx fix python linter in convert-hf-to-gguf.py

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    0921033 View commit details
    Browse the repository at this point in the history

  7. llama: support dbrx fix norm type

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    e4f8ee4 View commit details
    Browse the repository at this point in the history

  8. dbrx: minor

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    a7f9a3e View commit details
    Browse the repository at this point in the history

  9. convert: dbrx: fix mixed up and down expert tensors

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    e3c1e81 View commit details
    Browse the repository at this point in the history

  10. convert: dbrx: fix mixed up and down expert tensors 

    llama: dbrx: review graph
    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    0a35f58 View commit details
    Browse the repository at this point in the history

  11. doc: dbrx: add the model as supported

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    c8e6f90 View commit details
    Browse the repository at this point in the history

  12. convert: dbrx: fix remove wrong ATTN_OUT_NORM tensor, add output laye… 

    …r mapping
    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    916b918 View commit details
    Browse the repository at this point in the history

  13. llama: dbrx: remove wrong attn output layer in model arch

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    03da419 View commit details
    Browse the repository at this point in the history

  14. scripts: get-wikitext-2 add unzip

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    76f266b View commit details
    Browse the repository at this point in the history

  15. llama: dbrx: no attention output layer

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    9c7dedb View commit details
    Browse the repository at this point in the history

  16. model: dbrx: fix missing embedding tensor, mix with output layer

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    fe80898 View commit details
    Browse the repository at this point in the history

  17. llama: dbrx: remove not existing condition on empty output layer

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    4f12a58 View commit details
    Browse the repository at this point in the history

  18. Merge remote-tracking branch 'origin/master' into hp/model/support-dbrx

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    6985629 View commit details
    Browse the repository at this point in the history

  19. llama: dbrx: remove unnecessary optional tensor on FFN_GATE_EXPS

    @phymbert
    phymbert committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    7e7cd53 View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2024

  1. llama: increase maximum experts allowed

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    52c4033 View commit details
    Browse the repository at this point in the history

  2. model: dbrx: convert add n_ff

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    06a59ab View commit details
    Browse the repository at this point in the history

  3. llama: dbrx: quantize fix n_attention_wv tensor name

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    305ac3b View commit details
    Browse the repository at this point in the history

  4. model: dbrx: convert fix tokenizer

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    b6522a9 View commit details
    Browse the repository at this point in the history

  5. llama: dbrx: quantize fix n_attention_wv tensor name

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    dccb012 View commit details
    Browse the repository at this point in the history

  6. model: convert-hf-to-gguf.py add _set_vocab_tiktoken gpt2 backed on l… 

    …lama.cpp
    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    61be4b9 View commit details
    Browse the repository at this point in the history

  7. model: convert-hf-to-gguf.py fix classname conflict with qwen2

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    1fb6d95 View commit details
    Browse the repository at this point in the history

  8. model: dbrx: convert-hf-to-gguf.py fix fix ftype missing, fix tensor … 

    …names does not suffix with .weight
    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    200ce21 View commit details
    Browse the repository at this point in the history

  9. model: dbrx: convert-hf-to-gguf.py add chat template

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    9e17dad View commit details
    Browse the repository at this point in the history

  10. llama: quantize: remove wrong look for tensor qkv name as it was badl… 

    …y missing the .weight suffix
    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    d7546fd View commit details
    Browse the repository at this point in the history

  11. model: dbrx: convert-hf-to-gguf.py fix 'token_embd.weight' has wrong … 

    …shape, fix special tokens
    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    3a9dc2e View commit details
    Browse the repository at this point in the history

  12. model: dbrx: convert-hf-to-gguf.py support python 3.8

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    8154617 View commit details
    Browse the repository at this point in the history

  13. llama: dbrx: no weight suffix in ffn_gate_exps, ffn_up_exps and ffn_d… 

    …own_exps. Output tensor not optional.
    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    2449ef4 View commit details
    Browse the repository at this point in the history

  14. llama: quantize: remove wrong look for tensor qkv name as it was badl… 

    …y missing the .weight suffix

model: dbrx: convert to gguf force experts tensors to have .weight suffix
    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    1bd9427 View commit details
    Browse the repository at this point in the history

  15. llama: dbrx: fix tensor qkv number of elements

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    e9987c6 View commit details
    Browse the repository at this point in the history

  16. model: dbrx: convert reshape expert tensors to 3D

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    d151d8f View commit details
    Browse the repository at this point in the history

  17. model: dbrx: convert experts to f16

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    f062b83 View commit details
    Browse the repository at this point in the history

  18. model: dbrx: fix tensor names mapping broken

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    dbfd591 View commit details
    Browse the repository at this point in the history

  19. model: dbrx: fix expert reshape

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    7dd84b0 View commit details
    Browse the repository at this point in the history

  20. model: dbrx: fix expert reshape

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    c9bddbf View commit details
    Browse the repository at this point in the history

  21. model: dbrx: fix again sic expert reshape

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    e2c9199 View commit details
    Browse the repository at this point in the history

  22. model: dbrx: weird fix expert reshape

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    50b4373 View commit details
    Browse the repository at this point in the history

  23. llama: dbrx: output norm dim

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    0ab1bae View commit details
    Browse the repository at this point in the history

  24. llama: dbrx: fix last normalization

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    830e46d View commit details
    Browse the repository at this point in the history

  25. llama: dbrx: revert

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    2897aa6 View commit details
    Browse the repository at this point in the history

  26. llama: dbrx: move norm2 after attention, fix build kv

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    993f836 View commit details
    Browse the repository at this point in the history

  27. llama: dbrx: fix build kv att out

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    b01b062 View commit details
    Browse the repository at this point in the history

  28. llama: dbrx: fix build kv att out tensor name

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    74e6d87 View commit details
    Browse the repository at this point in the history

  29. llama: dbrx: hardcode nn.LayerNorm epsilon

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    f8f97e7 View commit details
    Browse the repository at this point in the history

  30. llama: dbrx: Try another rope type

    @phymbert
    phymbert committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    71f9e47 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2024

  1. llama: dbrx: fix k scale

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    52c6276 View commit details
    Browse the repository at this point in the history

  2. llama: dbrx: move norm epsilon to convert. Fix missing normalization.

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    8e22688 View commit details
    Browse the repository at this point in the history

  3. llama: dbrx: rename tensor to actual meaning. Fix normalization in gr… 

    …aph. Permute expert tensors to the llama.cpp layout
    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    35dce3e View commit details
    Browse the repository at this point in the history

  4. llama: dbrx: convert remove previous reverse

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    506cc2e View commit details
    Browse the repository at this point in the history

  5. llama: dbrx: load norm eps in hparams

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    eb0847e View commit details
    Browse the repository at this point in the history

  6. llama: dbrx: fix experts tensor layout

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    81f308a View commit details
    Browse the repository at this point in the history

  7. model: dbrx: convert-hf-to-gguf.py fix experts tensors shapes

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    21fb24a View commit details
    Browse the repository at this point in the history

  8. llama: factorize moe graph implementation between grok, mixtral and dbrx

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    f20c04f View commit details
    Browse the repository at this point in the history

  9. model: dbrx convert permute experts directly torch, log shape

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    48909ed View commit details
    Browse the repository at this point in the history

  10. llama: dbrx: fix experts 3D tensor layout (again)

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    18a84fe View commit details
    Browse the repository at this point in the history

  11. llama: dbrx: fix experts 3D tensor layout (again)

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    9968952 View commit details
    Browse the repository at this point in the history

  12. llama: dbrx: document changes, permute only FFN_DOWN_EXPS. Add a chec… 

    …k for ftype
    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    e66f1e3 View commit details
    Browse the repository at this point in the history

  13. llama: dbrx: rename layer_out_norm to attn_out_norm

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    f30a73b View commit details
    Browse the repository at this point in the history

  14. llama: dbrx: first add the residuals and then do the norm

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    ea8b58c View commit details
    Browse the repository at this point in the history

  15. model: dbrx: convert fix mixed ffn_gate_exps and ffn_down_exps

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    55943a2 View commit details
    Browse the repository at this point in the history

  16. llama: dbrx: fix ggml context of the attention outputs weight

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    c7b9a2e View commit details
    Browse the repository at this point in the history

  17. gguf-py: revert spaces

    @phymbert
    phymbert committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    ac82aa0 View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. gguf-py: dbrx: reverse again the MOE tensors mapping: 

        layer.ffn_up_exps   -> Up-projection weights (w1)
    layer.ffn_gate_exps -> Gating weights (v1)
    layer.ffn_down_exps -> Down-projection weights (w2)
    @phymbert
    phymbert committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    ac75fbd View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into hp/model/support-dbrx

    @phymbert
    phymbert committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    e5631cf View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. Merge remote-tracking branch 'origin/master' into hp/model/support-dbrx

    @phymbert
    phymbert committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    6f813dc View commit details
    Browse the repository at this point in the history

  2. llama: dbrx: use the MOE naming convention for model type

    @phymbert
    phymbert committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    74529e5 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. Merge remote-tracking branch 'origin/master' into hp/model/support-dbrx

    @phymbert
    phymbert committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    06527c6 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. model: convert-hf-to-gguf.py remove tiktoken

    @phymbert
    phymbert committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    fc89fee View commit details
    Browse the repository at this point in the history

  2. Is silu activation function applied to MODEL_TENSOR.FFN_GATE_EXP here… 

    …? If so, we must change this to w1 for DBRX. Each expert in DBRX has 3 linear layers: w1, v1 and w2. For an input tensor x, output from the expert layer would be (silu(x.w1_t) * x.v1_t) . w2_t). Same math is also used in mixtral, only difference being DBRX uses v1 instead of w3 in mixtral.

Co-authored-by: Megha Agarwal <[email protected]>
    @phymbert @megha95
    phymbert and megha95 committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    bdc4efe View commit details
    Browse the repository at this point in the history

  3. Is silu activation function applied to MODEL_TENSOR.FFN_GATE_EXP here… 

    …? If so, we must change this to w1 for DBRX. Each expert in DBRX has 3 linear layers: w1, v1 and w2. For an input tensor x, output from the expert layer would be (silu(x.w1_t) * x.v1_t) . w2_t). Same math is also used in mixtral, only difference being DBRX uses v1 instead of w3 in mixtral.

Co-authored-by: Megha Agarwal <[email protected]>
    @phymbert @megha95
    phymbert and megha95 committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    542585f View commit details
    Browse the repository at this point in the history

  4. Wrong input was being fed to moe layer. This needs to be corrected 

    Co-authored-by: Megha Agarwal <[email protected]>
    @phymbert @megha95
    phymbert and megha95 committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    ecbfb1b View commit details
    Browse the repository at this point in the history

  5. eval-callback: also print last n elements of each dimension

    @phymbert
    phymbert committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    647a11b View commit details
    Browse the repository at this point in the history

  6. minor spaces

    @phymbert
    phymbert committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    03bdc36 View commit details
    Browse the repository at this point in the history

  7. convert: update comment of MOE tensors mapping

    @phymbert
    phymbert committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    8e6758f View commit details
    Browse the repository at this point in the history

  8. llama: rename build_moe to build_moe_ffn and fix grok is using gelu i… 

    …nstead of silu.

Do not pass too much time on this function as it will be replaced in #6505
    @phymbert
    phymbert committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    f1256dc View commit details
    Browse the repository at this point in the history

  9. convert-hf-to-gguf.py: fix python linter

    @phymbert
    phymbert committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    e517585 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2024

  1. minor: fix indent in llama_build_graph

    @phymbert
    phymbert committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    9f77484 View commit details
    Browse the repository at this point in the history