Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support I/O with text and token ids #79

Closed
JoeZijunZhou opened this issue May 8, 2024 · 2 comments
Closed

Support I/O with text and token ids #79

JoeZijunZhou opened this issue May 8, 2024 · 2 comments
Assignees

Comments

@JoeZijunZhou
Copy link
Collaborator

  • Customer request: We use multiple languages for clients and cannot implement detokenization in each one. Need to have server-side detokenization support.
@kiratp
Copy link

kiratp commented May 12, 2024

Just chiming in here that the customer quoted is us :). The main challenger is that we have clients in multiple languages that don’t always have tokenizer implementations readily available. Every other prominent model server does detokenization, hence the request.

Doesn’t hurt that there are so many CPU cores on the TPU VMs that are mostly idle during inference anyway.

Thanks @JoeZijunZhou !

@JoeZijunZhou
Copy link
Collaborator Author

Resolved this issue in #78 . It's available in main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants