onnxruntime-gpu but GPU dependencies are not loaded

[SuperSecureHuman]

System Info

Optimum 1.13.1
Python 3.9


pip list | grep onnx                                                                                                                                                                                                                                          
onnx                      1.14.1
onnxruntime-gpu           1.16.0

Who can help?

@JingyaHuang @ech

Reproduction (minimal, reproducible, runnable)

Trying to use ORT with CUDA raises error -

onnxruntime-gpu is installed, but GPU dependencies are not loaded. It is likely there is a conflicting install between onnxruntime and onnxruntime-gpu. Please install only onnxruntime-gpu in order to use CUDAExecutionProvider."

While

Clearly, only ort-gpu is installed

Same issue can be reproduced in colab

%pip install optimum[onnxruntime-gpu]

ORTModelForSeq2SeqLM.from_pretrained(.... providers=["CUDAExecutionProvider"], use_io_binding=True)

Expected behavior

No error, expecting to run ort pipeline on GPU.

Workaround

Currently I am commenting out

optimum/optimum/onnxruntime/utils.py

Line 214 in 7fc27f6

def validate_provider_availability(provider: str):

Now the pipeline runs on GPU without any issues

[fxmarty]

Thank you @SuperSecureHuman, I can reproduce the issue. It appears onnxruntime-gpu==1.16.0 does not fill _ld_preload.py like it used to. I will patch!

[SuperSecureHuman]

I am not very familiar with onnx internals.

But from my understanding, if something is there in the available providers, it should work right?

Why is the check needed?

[fxmarty]

The check is "needed" because ORT falls back to CPUExecutionProvider without any error in case the CUDA / TRT install is wrong (or in case onnxruntime was installed after onnxruntime-gpu), see: https://onnxruntime.ai/docs/execution-providers/#use-execution-providers. We had a few issues where users were confused because of this (e.g. ORT was using CPU EP although specifying CUDAExecutionProvider, or failing although onnxruntime-gpu was installed due to a double onnxruntime/onnxruntime-gpu install, etc.) so I implemented hard checks with meaningful errors.

For the record, here is how onnxruntime/capi/_ld_preload.py used to look like with onnxruntime-gpu<=1.15.1 (while empty for onnxruntime). It appears ORT 1.16.0 changed this.

Fixed in #1403, will publish a patch today!

[SuperSecureHuman]

Got it!

Thanks!