Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added cleaned configuration properties for tokenizer with serialization - improve tokenization of XLM #1092

Merged
merged 12 commits into from
Aug 30, 2019
Merged

Added cleaned configuration properties for tokenizer with serialization - improve tokenization of XLM #1092

merged 12 commits into from
Aug 30, 2019

Commits on Aug 23, 2019

  1. Tokenization behave the same as original XLM proprocessing for most l… 

    …anguages except zh, ja and th; Change API to allow specifying language in `tokenize`
    @shijie-wu
    shijie-wu committed Aug 23, 2019
    Configuration menu
    Copy the full SHA
    436ce07 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2019

  1. Add custom tokenizer for zh and ja

    @shijie-wu
    shijie-wu committed Aug 24, 2019
    Configuration menu
    Copy the full SHA
    e85123d View commit details
    Browse the repository at this point in the history

  2. Add use_lang_emb to config

    @shijie-wu
    shijie-wu committed Aug 24, 2019
    Configuration menu
    Copy the full SHA
    f1b0187 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2019

  1. add kwargs to base encode function

    @thomwolf
    thomwolf committed Aug 27, 2019
    Configuration menu
    Copy the full SHA
    a175a9d View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2019

  1. Match order of casing in OSS XLM; Improve document; Clean up dependency

    @shijie-wu
    shijie-wu committed Aug 28, 2019
    Configuration menu
    Copy the full SHA
    ca4baf8 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2019

  1. Added option to setup pretrained tokenizer arguments

    @thomwolf
    thomwolf committed Aug 30, 2019
    Configuration menu
    Copy the full SHA
    82462c5 View commit details
    Browse the repository at this point in the history

  2. adding 17 and 100 xlm models

    @thomwolf
    thomwolf committed Aug 30, 2019
    Configuration menu
    Copy the full SHA
    8678ff8 View commit details
    Browse the repository at this point in the history

  3. adding xlm 17 and 100 models and config on aws

    @thomwolf
    thomwolf committed Aug 30, 2019
    Configuration menu
    Copy the full SHA
    3871b8a View commit details
    Browse the repository at this point in the history

  4. saving and reloading tokenizer configurations

    @thomwolf
    thomwolf committed Aug 30, 2019
    Configuration menu
    Copy the full SHA
    88111de View commit details
    Browse the repository at this point in the history

  5. added test and debug tokenizer configuration serialization

    @thomwolf
    thomwolf committed Aug 30, 2019
    Configuration menu
    Copy the full SHA
    69da972 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into xlm-tokenization

    @thomwolf
    thomwolf authored Aug 30, 2019
    Configuration menu
    Copy the full SHA
    cd65c41 View commit details
    Browse the repository at this point in the history

  7. fix tokenizers serialization

    @thomwolf
    thomwolf committed Aug 30, 2019
    Configuration menu
    Copy the full SHA
    7044ed6 View commit details
    Browse the repository at this point in the history