Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

llama : improve BPE pre-processing + LLaMA 3 and Deepseek support #6920

Merged
merged 61 commits into from
Apr 29, 2024
Merged

llama : improve BPE pre-processing + LLaMA 3 and Deepseek support #6920

merged 61 commits into from
Apr 29, 2024

Commits on Apr 26, 2024

  1. merged the changes from deepseeker models to main branch

    @jaggzh @ggerganov
    jaggzh authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    6fbab2d View commit details
    Browse the repository at this point in the history

  2. Moved regex patterns to unicode.cpp and updated unicode.h

    @dragnil1 @ggerganov
    dragnil1 authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    d2cfc22 View commit details
    Browse the repository at this point in the history

  3. Moved header files

    @dragnil1 @ggerganov
    dragnil1 authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    54f93eb View commit details
    Browse the repository at this point in the history

  4. Resolved issues

    @dragnil1 @ggerganov
    dragnil1 authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    1c924e4 View commit details
    Browse the repository at this point in the history

  5. added and refactored unicode_regex_split and related functions

    @dragnil1 @ggerganov
    dragnil1 authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    4056dc5 View commit details
    Browse the repository at this point in the history

  6. Updated/merged the deepseek coder pr

    @jaggzh @ggerganov
    jaggzh authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    c8e7d95 View commit details
    Browse the repository at this point in the history

  7. Refactored code

    @dragnil1 @ggerganov
    dragnil1 authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    4c3e882 View commit details
    Browse the repository at this point in the history

  8. Adding unicode regex mappings

    @dragnil1 @ggerganov
    dragnil1 authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    a5710a4 View commit details
    Browse the repository at this point in the history

  9. Adding unicode regex function

    @dragnil1 @ggerganov
    dragnil1 authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    7e308ed View commit details
    Browse the repository at this point in the history

  10. Added needed functionality, testing remains

    @dragnil1 @ggerganov
    dragnil1 authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    feeaf4f View commit details
    Browse the repository at this point in the history

  11. Fixed issues

    @dragnil1 @ggerganov
    dragnil1 authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    7535803 View commit details
    Browse the repository at this point in the history

  12. Fixed issue with gpt2 regex custom preprocessor

    @dragnil1 @ggerganov
    dragnil1 authored and ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    36d9832 View commit details
    Browse the repository at this point in the history

  13. unicode : fix? unicode_wstring_to_utf8

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    06d3e69 View commit details
    Browse the repository at this point in the history

  14. lint : fix whitespaces

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    c56e19d View commit details
    Browse the repository at this point in the history

  15. tests : add tokenizer tests for numbers

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    7a44e44 View commit details
    Browse the repository at this point in the history

  16. unicode : remove redundant headers

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    d999cf6 View commit details
    Browse the repository at this point in the history

  17. tests : remove and rename tokenizer test scripts

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    aeafb43 View commit details
    Browse the repository at this point in the history

  18. tests : add sample usage

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    e1b2bf7 View commit details
    Browse the repository at this point in the history

  19. gguf-py : reader prints warnings on duplicate keys

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    ed42711 View commit details
    Browse the repository at this point in the history

  20. llama : towards llama3 tokenization support (wip)

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    4907e41 View commit details
    Browse the repository at this point in the history

  21. unicode : shot in the dark to fix tests on Windows

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    e8c206b View commit details
    Browse the repository at this point in the history

  22. unicode : first try custom implementations

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    e989176 View commit details
    Browse the repository at this point in the history

  23. Merge branch 'master' into gg/bpe-preprocess

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    e3f6dc7 View commit details
    Browse the repository at this point in the history

  24. convert : add "tokenizer.ggml.pre" GGUF KV (wip)

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    9b4d63a View commit details
    Browse the repository at this point in the history

  25. llama : use new pre-tokenizer type

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    43e12ce View commit details
    Browse the repository at this point in the history

  26. convert : fix pre-tokenizer type writing

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    1b9b79d View commit details
    Browse the repository at this point in the history

  27. lint : fix

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    8791e94 View commit details
    Browse the repository at this point in the history

  28. make : add test-tokenizer-0-llama-v3

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    a774d70 View commit details
    Browse the repository at this point in the history

  29. wip

    @ggerganov
    ggerganov committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    c160818 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2024

  1. models : add llama v3 vocab file

    @ggerganov
    ggerganov committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    96965f6 View commit details
    Browse the repository at this point in the history

  2. llama : adapt punctuation regex + add llama 3 regex

    @ggerganov
    ggerganov committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    ad92983 View commit details
    Browse the repository at this point in the history

  3. minor

    @ggerganov
    ggerganov committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    4434c9d View commit details
    Browse the repository at this point in the history

  4. unicode : set bomb

    @ggerganov
    ggerganov committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    a22645c View commit details
    Browse the repository at this point in the history

  5. unicode : set bomb

    @ggerganov
    ggerganov committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    2affd0b View commit details
    Browse the repository at this point in the history

  6. unicode : always use std::wregex

    @ggerganov
    ggerganov committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    ce5485a View commit details
    Browse the repository at this point in the history

  7. unicode : support \p{N}, \p{L} and \p{P} natively

    @ggerganov
    ggerganov committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    91eaa41 View commit details
    Browse the repository at this point in the history

  8. unicode : try fix windows

    @ggerganov
    ggerganov committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    581c4a0 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2024

  1. unicode : category support via std::regex

    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    b97add5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into gg/bpe-preprocess 

    ggml-ci
    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    d63cc90 View commit details
    Browse the repository at this point in the history

  3. unicode : clean-up

    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    e972e6c View commit details
    Browse the repository at this point in the history

  4. unicode : simplify

    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    ee6d1b3 View commit details
    Browse the repository at this point in the history

  5. convert : add convert-hf-to-gguf-update.py 

    ggml-ci
    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    7642973 View commit details
    Browse the repository at this point in the history

  6. lint : update

    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    4e3e6d8 View commit details
    Browse the repository at this point in the history

  7. convert : add falcon 

    ggml-ci
    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    1c888eb View commit details
    Browse the repository at this point in the history

  8. unicode : normalize signatures

    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    1545550 View commit details
    Browse the repository at this point in the history

  9. lint : fix

    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    491f233 View commit details
    Browse the repository at this point in the history

  10. lint : fix

    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    e8dd4a1 View commit details
    Browse the repository at this point in the history

  11. convert : remove unused functions

    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    02fd977 View commit details
    Browse the repository at this point in the history

  12. convert : add comments

    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    0f9058c View commit details
    Browse the repository at this point in the history

  13. convert : exercise contractions 

    ggml-ci
    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    7808150 View commit details
    Browse the repository at this point in the history

  14. lint : fix

    @ggerganov
    ggerganov committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    7b1210f View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. cmake : refactor test targets

    @ggerganov
    ggerganov committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    ef4cca9 View commit details
    Browse the repository at this point in the history

  2. tests : refactor vocab tests 

    ggml-ci
    @ggerganov
    ggerganov committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    43708d2 View commit details
    Browse the repository at this point in the history

  3. tests : add more vocabs and tests 

    ggml-ci
    @ggerganov
    ggerganov committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    c68d259 View commit details
    Browse the repository at this point in the history

  4. unicode : cleanup

    @ggerganov
    ggerganov committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    af05268 View commit details
    Browse the repository at this point in the history

  5. scripts : ignore new update script in check-requirements.sh

    @ggerganov
    ggerganov committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    c21ab18 View commit details
    Browse the repository at this point in the history

  6. models : add phi-3, mpt, gpt-2, starcoder

    @ggerganov
    ggerganov committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    120cf37 View commit details
    Browse the repository at this point in the history

  7. tests : disable obsolete 

    ggml-ci
    @ggerganov
    ggerganov committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    9a7d430 View commit details
    Browse the repository at this point in the history

  8. tests : use faster bpe test 

    ggml-ci
    @ggerganov
    ggerganov committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    6d6ce93 View commit details
    Browse the repository at this point in the history

  9. llama : more prominent warning for old BPE models

    @ggerganov
    ggerganov committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    3202676 View commit details
    Browse the repository at this point in the history

  10. tests : disable test-tokenizer-1-bpe due to slowness 

    ggml-ci
    @ggerganov
    ggerganov committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    80cb312 View commit details
    Browse the repository at this point in the history