Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support the Unigram tokenizer kind from sentencepiece library #811

Merged
merged 8 commits into from
Sep 19, 2024
Merged

Support the Unigram tokenizer kind from sentencepiece library #811

merged 8 commits into from
Sep 19, 2024

Conversation

wenbingl
Copy link
Member

@wenbingl wenbingl commented Sep 18, 2024 •
edited
Loading

  1. Add a new type of tokenizer to support the Unigram tokenizer kind.
  2. Refine bpe tokenizer loading code.

@wenbingl wenbingl marked this pull request as ready for review September 19, 2024 17:10
@wenbingl wenbingl requested a review from a team as a code owner September 19, 2024 17:10
@wenbingl wenbingl merged commit 176c1d0 into main Sep 19, 2024
46 checks passed
@wenbingl wenbingl deleted the ugm branch September 19, 2024 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sayanshaw24 sayanshaw24 sayanshaw24 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@wenbingl @sayanshaw24