Ray does not handle MIG devices

[maxclaey]

What is the problem?

Today we discovered an issue with a ray deployment on a DGX A100. NVIDIAs new Ampere cards support MIG (multi instance GPUs) where a physical GPU is split into multiple virtual GPUs, that each can be used as a normal CUDA device. However, in /proc/driver/nvidia/gpus only the physical GPUs show up. So in case you have a single 40GB GPU with 4 10GB MIG devices, you will only see 1 GPU, while in fact there are 4 (virtual ones).

Furthermore, these MIG devices can no longer be specified with CUDA_VISIBLE_DEVICES using a simple ID, but the actual UUIDs should be used.

I had a quick shot at writing a script that can fetch all MIG devices and all regular devices not in MIG mode (see attachment). Maybe it could be interesting to incorporate something like this to make ray able to handle NVIDIAs latest hardware? Thanks in advance for considering

gpu_uuid_fetching.txt (File uploaded as .txt file instead of .py due to GitHub restrictions)

Reproduction (REQUIRED)

Check the available GPU resources on a system with Ampere GPUs and MIG enabled. The number of GPU resources will reflect the number of physical GPUs, rather than the amount of MIG devices.

• [X ] I have verified my script runs in a clean environment and reproduces the issue.
• [X ] I have verified the issue also occurs with the latest wheels.

[richardliaw]

This seems quite cool! How did you get access to the a100?

[maxclaey]

We have one available at Robovision :)

[richardliaw]

@ericl @wuisawesome @simon-mo we probably want to have some way of plugging in accelerator detectors. They would set the correct environment variables, and handle resources similar to how GPUs are handled.

This will allow us to easily support TPUs, new hardware, etc.

[PidgeyBE]

FYI: If we set CUDA_VISIBLE_DEVICES before starting raylet, and set --num_gpus to the number of MIGs, it works!

[dominicshanshan]

FYI: If we set CUDA_VISIBLE_DEVICES before starting raylet, and set --num_gpus to the number of MIGs, it works!

I tried start 4 MIG in one docker, then start raylet, nvidia-smi can identify 4MIG, but ray can only run one MIG

[DuoblaK]

What is the problem?

Today we discovered an issue with a ray deployment on a DGX A100. NVIDIAs new Ampere cards support MIG (multi instance GPUs) where a physical GPU is split into multiple virtual GPUs, that each can be used as a normal CUDA device. However, in /proc/driver/nvidia/gpus only the physical GPUs show up. So in case you have a single 40GB GPU with 4 10GB MIG devices, you will only see 1 GPU, while in fact there are 4 (virtual ones).

Furthermore, these MIG devices can no longer be specified with CUDA_VISIBLE_DEVICES using a simple ID, but the actual UUIDs should be used.

I had a quick shot at writing a script that can fetch all MIG devices and all regular devices not in MIG mode (see attachment). Maybe it could be interesting to incorporate something like this to make ray able to handle NVIDIAs latest hardware? Thanks in advance for considering

gpu_uuid_fetching.txt (File uploaded as .txt file instead of .py due to GitHub restrictions)

Reproduction (REQUIRED)

Check the available GPU resources on a system with Ampere GPUs and MIG enabled. The number of GPU resources will reflect the number of physical GPUs, rather than the amount of MIG devices.

• [X ] I have verified my script runs in a clean environment and reproduces the issue.
• [X ] I have verified the issue also occurs with the latest wheels.

Hello，has this question solved for you?
I have met the same problem.
And I tried ray 2.0.0 still exits this problem.
Plus, if my mig device exits on the same gpu device, ray can see different mig device, else if my mig exits on the same gpu device, then ray can see only one mig device.

[ericl]

As a workaround, I believe the following should work (referencing #12413 (comment)):

1. Set CUDA_VISIBLE_DEVICES=uuid1,uuid2 prior to "ray start".
2. Also set --num-gpus explicitly in "ray start".

e.g., CUDA_VISIBLE_DEVICES=uuid1,uuid2,uuid3 ray start --num-gpus=3.

Should we auto-detect MIG devices by default? @amogkam

[XuehaiPan]

I think we should use libcuda.so to detect CUDA devices, see also #17914 (comment).

• [RLLIB] Windows WSL GPU Not Detected  #17914

Currently, CUDA program can only enumerate one MIG device, see https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html#cuda-visible-devices. Ray should read all available MIG devices then set CUDA_VISIBLE_DEVICES for each remote actor.

[DuoblaK]

As a workaround, I believe the following should work (referencing #12413 (comment)):

1. Set CUDA_VISIBLE_DEVICES=uuid1,uuid2 prior to "ray start".
2. Also set --num-gpus explicitly in "ray start".

e.g., CUDA_VISIBLE_DEVICES=uuid1,uuid2,uuid3 ray start --num-gpus=3.

Should we auto-detect MIG devices by default? @amogkam

Thanks for you reply, and I tried as you said, it really worked in the condition in which pod running on single node in k8s pod, multi-nodes still has problem and I am working on it, it should be working with the same way.
Thanks again for your reply, it helps me a lot.

[huaiyizhao]

Hello, any updates on this thread besides the workaround by @ericl?

[peterghaddad]

@XuehaiPan @amogkam Any updates on this?