Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NllbTokenizer] refactor with added tokens decoder #27717

Merged
merged 36 commits into from
Feb 13, 2024
Merged

[NllbTokenizer] refactor with added tokens decoder #27717

merged 36 commits into from
Feb 13, 2024

Commits on Nov 27, 2023

  1. refactor with addedtokens decoder

    @ArthurZucker
    ArthurZucker committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    76dd325 View commit details
    Browse the repository at this point in the history

  2. style

    @ArthurZucker
    ArthurZucker committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    862d6d1 View commit details
    Browse the repository at this point in the history

  3. get rid of lang code to id

    @ArthurZucker
    ArthurZucker committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    0ae401f View commit details
    Browse the repository at this point in the history

  4. style

    @ArthurZucker
    ArthurZucker committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    b230b6b View commit details
    Browse the repository at this point in the history

  5. keep some things for BC

    @ArthurZucker
    ArthurZucker committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    329569f View commit details
    Browse the repository at this point in the history

  6. update tests

    @ArthurZucker
    ArthurZucker committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    ee08b8d View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2023

  1. add the mask token at the end of the vocab

    @ArthurZucker
    ArthurZucker committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    d5b3195 View commit details
    Browse the repository at this point in the history

  2. nits

    @ArthurZucker
    ArthurZucker committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    086000a View commit details
    Browse the repository at this point in the history

  3. nits

    @ArthurZucker
    ArthurZucker committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    23aa840 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' of github.com:huggingface/transformers into refac… 

    …tor-nllb
    @ArthurZucker
    ArthurZucker committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    dbd80f6 View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2024

  1. Merge branch 'main' of github.com:huggingface/transformers into refac… 

    …tor-nllb
    @ArthurZucker
    ArthurZucker committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    5ec00d2 View commit details
    Browse the repository at this point in the history

  2. fix final tests

    @ArthurZucker
    ArthurZucker committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    8a14c0b View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2024

  1. Merge branch 'main' of github.com:huggingface/transformers into refac… 

    …tor-nllb
    @ArthurZucker
    ArthurZucker committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    a77609f View commit details
    Browse the repository at this point in the history

  2. style

    @ArthurZucker
    ArthurZucker committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    f79abb7 View commit details
    Browse the repository at this point in the history

  3. nits

    @ArthurZucker
    ArthurZucker committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    f74b89c View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2024

  1. Update src/transformers/models/nllb/tokenization_nllb_fast.py 

    Co-authored-by: amyeroberts <[email protected]>
    @ArthurZucker @amyeroberts
    ArthurZucker and amyeroberts authored Jan 18, 2024
    Configuration menu
    Copy the full SHA
    337f068 View commit details
    Browse the repository at this point in the history

  2. nits

    @ArthurZucker
    ArthurZucker committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    c79212e View commit details
    Browse the repository at this point in the history

  3. style?

    @ArthurZucker
    ArthurZucker committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    0094394 View commit details
    Browse the repository at this point in the history

  4. Update src/transformers/convert_slow_tokenizer.py

    @ArthurZucker
    ArthurZucker authored Jan 18, 2024
    Configuration menu
    Copy the full SHA
    ea9a78e View commit details
    Browse the repository at this point in the history

  5. make it a tad bit more custom

    @ArthurZucker
    ArthurZucker committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    424c8e4 View commit details
    Browse the repository at this point in the history

  6. ruff please stop 

    Co-Authored by avidale

<[email protected]>
    @ArthurZucker
    ArthurZucker committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    eac3947 View commit details
    Browse the repository at this point in the history

  7. Update 

    Co-authored-by: avidale
<[email protected]>
    @ArthurZucker
    ArthurZucker committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    6bf2631 View commit details
    Browse the repository at this point in the history

  8. Update 

    Co-authored-by: avidale <[email protected]>
    @ArthurZucker
    ArthurZucker committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    fa8ab61 View commit details
    Browse the repository at this point in the history

  9. oupts

    @ArthurZucker
    ArthurZucker committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    db35a5d View commit details
    Browse the repository at this point in the history

  10. ouft

    @ArthurZucker
    ArthurZucker committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    0d19d4c View commit details
    Browse the repository at this point in the history

  11. nites

    @ArthurZucker
    ArthurZucker committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    3e01ea7 View commit details
    Browse the repository at this point in the history

  12. test

    @ArthurZucker
    ArthurZucker committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    ea6906c View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. Merge branch 'main' of github.com:huggingface/transformers into refac… 

    …tor-nllb
    @ArthurZucker
    ArthurZucker committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    dc56b4a View commit details
    Browse the repository at this point in the history

  2. fix the remaining failing tests

    @ArthurZucker
    ArthurZucker committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    4f21f5e View commit details
    Browse the repository at this point in the history

  3. style

    @ArthurZucker
    ArthurZucker committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    0f44c39 View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2024

  1. fix failing test

    @ArthurZucker
    ArthurZucker committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    a038369 View commit details
    Browse the repository at this point in the history

  2. ficx other test

    @ArthurZucker
    ArthurZucker committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    7172ad7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' of github.com:huggingface/transformers into refac… 

    …tor-nllb
    @ArthurZucker
    ArthurZucker committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    93c02de View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2024

  1. temp dir + test the raw init

    @ArthurZucker
    ArthurZucker committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    f2ba78f View commit details
    Browse the repository at this point in the history

  2. update test

    @ArthurZucker
    ArthurZucker committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    5b08ee0 View commit details
    Browse the repository at this point in the history

  3. style

    @ArthurZucker
    ArthurZucker committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    ed86fa6 View commit details
    Browse the repository at this point in the history