[Tokenizer] Fix decode output with space in decode_token

[DrownFish19]

PR types

Bug fixes

PR changes

Others

Description

Fix decode output with space in decode_token.
For example, new_text = " 123"(space + 123), and prefix_text = " "(space),the clean_up_tokenization_spaces will make new_text to "123".

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 53.96%. Comparing base (56d293d) to head (0cd1c52).
Report is 77 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #9010      +/-   ##
===========================================
- Coverage    54.08%   53.96%   -0.13%     
===========================================
  Files          650      652       +2     
  Lines       103915   104929    +1014     
===========================================
+ Hits         56200    56621     +421     
- Misses       47715    48308     +593     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ZHUI]

LGTM