Skip to content

Releases: yishn/chinese-tokenizer

chinese-tokenizer v2.4.0

05 Jun 13:01
Compare
Choose a tag to compare
  • Add position property to tokens

chinese-tokenizer v2.1.0

29 May 01:41
Compare
Choose a tag to compare
  • Handle tokenization of non-Chinese text gracefully