Make sure integrated GPU is used

[patins1]

Hi, I have a notebook with integrated Intel Iris Xe graphics and no other external GPUs.
How can i make sure that that one is picked up in calculations and not the cpu?

Configuring with .optDevices(new Device[] { Device.gpu() }) results in this exception:
ai.djl.engine.EngineException: MXNet engine call failed: MXNetError: Compile with USE_CUDA=1 to enable GPU usage

[frankfliu]

In DJL, when we refer to GPU it actually means CUDA. You machine doesn't have CUDA installed, so you won't be able to use GPU device.

[patins1]

Now I wonder, as DJL makes such a nice job to install all dependencies like mxnet when invoking the engine, whether DJL can also install CUDA automatically as soon as my code tries to use the GPU device?

Otherwise, how would I install CUDA manually?

[frankfliu]

@patins1

You can download CUDA: https://developer.nvidia.com/cuda-toolkit
You also need install CUDA driver that matches your Nvidia graphic card: https://www.nvidia.com/Download/index.aspx

I don't think it's feasible for DJL to install CUDA:

1. CUDA is more that a library, it include a driver. Run installer requires admin permission, which DJL app usually won't have.
2. CUDA driver installation usually requires reboot the machine, as a Java library we won't be able to do that.
3. Currently DJL relies on CUDA driver to detect the type of GPU (sm architecture) and version if CUDA library, DJL won't be able to know which driver/version of CUDA to install on the system.

[patins1]

I installed CUDA, and i do not get the EngineException anymore, that is good. However, Engine.getInstance().getGpuCount() still yields 0 though I have one integrated GPU, how can this be?

[frankfliu]

@patins1

That means CUDA driver did not detect you have physical GPU in your system.

You mentioned you are using Intel graphic card, CUDA only works on Nvidia graphic card.
You can run nvidia-smi to show your graphic information.

[patins1]

Hm, does Engine.getInstance().getGpuCount() include integrated GPUs?

[frankfliu]

No, only CUDA GPU can be used for computation, all other graphic cards can only be used for graphic rendering.
See: Which GPUs support CUDA?

[patins1]

I bought my current computer as Intel put as one advantage of the Intel Iris XE the "Intel® Deep Learning Boost (Intel® DL Boost)" feature. Does it mean that DJL cannot make use of it ? :(

"A new set of embedded processor technologies designed to accelerate AI deep learning use cases. It extends Intel AVX-512 with a new Vector Neural Network Instruction (VNNI) that significantly increases deep learning inference performance over previous generations."

[frankfliu]

AVX-512 with (VNNI) is a CPU architecture, latest MXNet, PyTorch and DLR engine should already support it.

You should get performance boost when you using CPU device with DJL. But compare to CUDA they are way slow.

[patins1]

Thanks for all the help, now my final question: how can can I execute my djl program in a modus that uses (a) the integrated GPU and in a modus that uses (b) the CPU alone without any GPU participation? Looking on the Performance tab in Windows Task Manager, the CPU is used 95% and the GPU only to 5%. I have updated all jars a few days ago, so it seems that MXNet is not utilizing the integrated GPU yet? Or maybe I have force DJL to download all dependencies again which it did only once when I used older jars. How could I do that?

[frankfliu]

DJL depends on underlying engines to use GPU. Unfortunately, none of none of DJL supported engine can use Intel GPUs (DLR might be able to leverage Intel GPU). See: https://stackoverflow.com/questions/64593792/how-to-make-intel-gpu-available-for-processing-through-pytorch

[patins1]

I just found out how to tweak the performance much more than using a GPU: Using PyTorch !!
MXNet took 1 1/2 hour to train my model, PyTorch takes 3.2 minutes !! (??)
I first thought it is because PyTorch found a way to use the integrated Iris GPU, however the Windows task manager says it uses only the CPU. My only other explanation is: Python is so slow that they had to back it by efficient C code, while MXNet uses .NET assemblies ?