Skip to content

Commit

Permalink
Update hugging_face.py (#5488)
Browse files Browse the repository at this point in the history
  • Loading branch information
dfokina authored Aug 3, 2023
1 parent 56cea8c commit 1f88cd1
Showing 1 changed file with 7 additions and 7 deletions.
14 changes: 7 additions & 7 deletions haystack/nodes/prompt/invocation_layer/hugging_face.py
Original file line number Diff line number Diff line change
Expand Up @@ -334,16 +334,16 @@ def _prepare_tokenizer(
self, model: Union[str, "PreTrainedModel"], hub_kwargs: Dict, model_kwargs: Optional[Dict] = None
) -> Union["PreTrainedTokenizer", "PreTrainedTokenizerFast", None]:
"""
this method prepares the tokenizer before passing it to transformers' pipeline, so that the instantiated pipeline
This method prepares the tokenizer before passing it to transformers' pipeline, so that the instantiated pipeline
object has a working tokenizer.
It basically check whether the pipeline method in the transformers library will load the tokenizer.
- If yes, None will be returned, because in this case, the pipeline is intelligent enough to load the tokenizer by itself
- If not, we will load the tokenizer and an tokenizer instance is returned
It checks whether the pipeline method in the transformers library will load the tokenizer.
- If yes, None will be returned, because in this case, the pipeline is intelligent enough to load the tokenizer by itself.
- If not, we will load the tokenizer and an tokenizer instance is returned.
:param model: the name or path of the underlying model
:hub_kwargs: keyword argument related to hugging face hub, including revision, trust_remote_code and use_auth_token
:model_kwargs: keyword arguments passed to the underlying model
:param model: The name or path of the underlying model.
:hub_kwargs: Keyword argument related to hugging face hub, including revision, trust_remote_code and use_auth_token.
:model_kwargs: Keyword arguments passed to the underlying model.
"""

if isinstance(model, str):
Expand Down

0 comments on commit 1f88cd1

Please sign in to comment.