What's the reason for not including the tokenizer in the repo?

[nicastelo]

Hi! I'm looking to write a ruby client and was wondering why not including the tokenizer in the repo instead of downloading it from the web. Is it because of licensing or similar?

Thanks!

[hzeng-otterai]

Same question here. When I use anthropic to work with langchain, every time when I count the tokens there are multiple network requests to Huggingface to download the tokenizer files.

2023-04-29 10:06:45.180 Starting new HTTPS connection (1): huggingface.co:443
2023-04-29 10:06:45.325 https://huggingface.co:443 "HEAD /gpt2/resolve/main/tokenizer_config.json HTTP/1.1" 404 0
2023-04-29 10:06:45.336 Starting new HTTPS connection (1): huggingface.co:443
2023-04-29 10:06:45.474 https://huggingface.co:443 "HEAD /gpt2/resolve/main/vocab.json HTTP/1.1" 200 0
2023-04-29 10:06:45.482 Starting new HTTPS connection (1): huggingface.co:443
2023-04-29 10:06:45.619 https://huggingface.co:443 "HEAD /gpt2/resolve/main/merges.txt HTTP/1.1" 200 0
2023-04-29 10:06:45.630 Starting new HTTPS connection (1): huggingface.co:443
2023-04-29 10:06:45.755 https://huggingface.co:443 "HEAD /gpt2/resolve/main/tokenizer.json HTTP/1.1" 200 0
2023-04-29 10:06:45.767 Starting new HTTPS connection (1): huggingface.co:443
2023-04-29 10:06:45.899 https://huggingface.co:443 "HEAD /gpt2/resolve/main/added_tokens.json HTTP/1.1" 404 0
2023-04-29 10:06:45.913 Starting new HTTPS connection (1): huggingface.co:443
2023-04-29 10:06:46.044 https://huggingface.co:443 "HEAD /gpt2/resolve/main/special_tokens_map.json HTTP/1.1" 404 0
2023-04-29 10:06:46.048 Starting new HTTPS connection (1): huggingface.co:443
2023-04-29 10:06:46.180 https://huggingface.co:443 "HEAD /gpt2/resolve/main/tokenizer_config.json HTTP/1.1" 404 0
2023-04-29 10:06:46.188 Starting new HTTPS connection (1): huggingface.co:443
2023-04-29 10:06:46.361 https://huggingface.co:443 "HEAD /gpt2/resolve/main/config.json HTTP/1.1" 200 0

[jenan-anthropic]

The new version of the SDK includes the tokenizer in the repo!