Please consider support the new jina rerank model

[saggit]

First thanks for the great work. infinity is what i found the most perfect framework to serve embedding models.

And Jina recently update a new model: Jina Reranker v2 for Agentic RAG: Ultra-Fast, Multilingual, Function-Calling & Code Search

And i tried to load it and get the errors below(i am not a expert on this field):

infinity_emb v2 --model-id /root/app/models/models--jinaai--jina-reranker-v2-base-multilingual --port 5029
INFO     2024-07-17 03:52:01,916 datasets INFO: PyTorch version 2.3.1 available.                                                                                             config.py:58
INFO:     Started server process [1371]
INFO:     Waiting for application startup.
INFO     2024-07-17 03:52:04,976 infinity_emb INFO: model=`/root/app/models/models--jinaai--jina-reranker-v2-base-multilingual` selected, using engine=`torch` and     select_model.py:57
         device=`None`
INFO     2024-07-17 03:52:05,841 sentence_transformers.cross_encoder.CrossEncoder INFO: Use pytorch device: cuda                                                      CrossEncoder.py:110
INFO     2024-07-17 03:52:08,623 infinity_emb INFO: Adding optimizations via Huggingface optimum.                                                                      acceleration.py:46
WARNING  2024-07-17 03:52:08,625 infinity_emb WARNING: BetterTransformer is not available for model: <class                                                            acceleration.py:57
         'transformers_modules.models--jinaai--jina-reranker-v2-base-multilingual.modeling_xlm_roberta.XLMRobertaForSequenceClassification'> Continue without
         bettertransformer modeling code.
INFO     2024-07-17 03:52:08,626 infinity_emb INFO: Switching to half() precision (cuda: fp16).                                                                               torch.py:68
INFO     2024-07-17 03:52:10,465 infinity_emb INFO: Getting timings for batch_size=32 and avg tokens per sentence=2                                                    select_model.py:80
                 2.15     ms tokenization
                 14.85    ms inference
                 0.01     ms post-processing
                 17.01    ms total
         embeddings/sec: 1881.69
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [44,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

.....

../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [597,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [597,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [597,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1289: indexSelectLargeIndex: block: [597,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.


ERROR:    Traceback (most recent call last):
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/starlette/routing.py", line 732, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/infinity_emb/infinity_server.py", line 59, in lifespan
    app.engine_array = AsyncEngineArray.from_args(engine_args_list)  # type: ignore
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/infinity_emb/engine.py", line 252, in from_args
    return cls(engines=tuple(engines))
                       ^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/infinity_emb/engine.py", line 64, in from_args
    engine = cls(**engine_args.to_dict(), _show_deprecation_warning=False)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/infinity_emb/engine.py", line 50, in __init__
    self._model, self._min_inference_t, self._max_inference_t = select_model(
                                                                ^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/infinity_emb/inference/select_model.py", line 82, in select_model
    loaded_engine.warmup(batch_size=engine_args.batch_size, n_tokens=512)
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/infinity_emb/transformer/abstract.py", line 167, in warmup
    return run_warmup(self, inp)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/infinity_emb/transformer/abstract.py", line 175, in run_warmup
    embed = model.encode_core(feat)
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/infinity_emb/transformer/crossencoder/torch.py", line 86, in encode_core
    out_features = self.model(**features, return_dict=True)["logits"]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/models--jinaai--jina-reranker-v2-base-multilingual/modeling_xlm_roberta.py", line 854, in forward
    outputs = self.roberta(
              ^^^^^^^^^^^^^
File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/models--jinaai--jina-reranker-v2-base-multilingual/modeling_xlm_roberta.py", line 664, in forward
    sequence_output = self.encoder(
                      ^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/vector-service/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/models--jinaai--jina-reranker-v2-base-multilingual/modeling_xlm_roberta.py", line 217, in forward
    hidden_states, indices, cu_seqlens, max_seqlen_in_batch = unpad_input(
                                                              ^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/models--jinaai--jina-reranker-v2-base-multilingual/xlm_padding.py", line 113, in unpad_input
    indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


ERROR:    Application startup failed. Exiting.

[michaelfeil]

Hi, please upvote this pr: https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual/discussions/18

infinity_emb v2 --model-id jinaai/jina-reranker-v2-base-multilingual --revision "refs/pr/18"

[saggit]

Thanks for reply. You are a hero and life saver. It works and I will upvote.

[michaelfeil]

Thanks for the kind words!