Added cleaned configuration properties for tokenizer with serialization - improve tokenization of XLM

[shijie-wu]

This PR improve the tokenization of XLM. It's mostly the same as the preprocessing in the original XLM. This PR also add use_lang_emb to config of XLM model, which makes adding the newly release XLM-17 & XLM-100 easier since both of them don't have language embedding.

Details on tokenization:

• Introduce API change: Changing XLMTokenizer.tokenize(self, text) to XLMTokenizer.tokenize(text, lang='en')
• New dependency:
  • sacremoses: port of Moses
• New optional dependencies:
  • pythainlp: Thai tokenizer
  • kytea: Japanese tokenizer, wrapper of KyTea (Need external C++ compilation), used by the newly release XLM-17 & XLM-100
  • jieba: Chinese tokenizer *

* XLM used Stanford Segmenter. However, the wrapper (nltk.tokenize.stanford_segmenter) are slow due to JVM overhead, and it will be deprecated. Jieba is a lot faster and pip-installable. But there is some mismatch with the Stanford Segmenter. A workaround could be having an argument to allow users to segment the sentence by themselves and bypass the segmenter. As a reference, I also include nltk.tokenize.stanford_segmenter in this PR.

Example of tokenization difference could be found here.

[thomwolf]

Thanks a lot for all this work, it's great!

I've made a few comments on things to update. Mostly that we are only going to add sacremoses as required dependencies and raise error messages for the others.

I need to do a few modifications up-stream as mentioned in the comments to make it easier here.
Will do it in another PR so you can have a look.

[codecov-io]

Codecov Report

Merging #1092 into master will increase coverage by 0.09%.
The diff coverage is 78.2%.

@@            Coverage Diff             @@
##           master    #1092      +/-   ##
==========================================
+ Coverage   79.61%   79.71%   +0.09%     
==========================================
  Files          42       42              
  Lines        6898     7010     +112     
==========================================
+ Hits         5492     5588      +96     
- Misses       1406     1422      +16

Impacted Files	Coverage Δ
pytorch_transformers/modeling_xlm.py	86.73% <100%> (+0.07%)	⬆️
pytorch_transformers/tokenization_bert.py	95.63% <100%> (+0.79%)	⬆️
pytorch_transformers/tokenization_utils.py	86.49% <100%> (+0.26%)	⬆️
pytorch_transformers/tokenization_xlm.py	83.4% <74.43%> (+0.33%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df9d6ef...3871b8a. Read the comment docs.

[thomwolf]

Hi @shijie-wu,
So I've taken advantage of this PR to add a clean mechanism to set, save and reload tokenizer configurations.
This should fix in particular a recurring issue mentioned in #1158 and #1026 (failing to reload the lower casing configuration of the tokenizer) but more generally this is essential now for XLM's more complex language configuration.
Hope you don't mind me highjacking the PR.

[thomwolf]

Ok I think this is good to go. Let's merge it.