Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix GPU Discovery. #4522

Merged
merged 3 commits into from
Aug 29, 2023
Merged

Fix GPU Discovery. #4522

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7ab01e7
Fix gpu output. Already tested on slurm
AndrewJGaut Aug 26, 2023
4b5f0be
Add more logging
AndrewJGaut Aug 26, 2023
48ea608
Merge branch 'master' into fix-gpu-discovery
wwwjn Aug 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion codalab/worker/docker_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,11 @@ def get_nvidia_devices(self, use_docker=True):
docker.errors.ImageNotFound if the CUDA image cannot be pulled
docker.errors.APIError if another server error occurs
"""
cuda_image = 'sulfurheron/nvidia-cuda:9.0-cudnn7-devel-ubuntu16.04-2018-06-08'

# Note: Do NOT update the NVIDIA image to use a CUDA version higher than
# that supported by the NLP machines. Otherwise, Slurm Batch Worker
# Manager will no longer function.
cuda_image = 'nvidia/cuda:11.5.2-base-ubuntu20.04'
epicfaace marked this conversation as resolved.
Show resolved Hide resolved
nvidia_command = 'nvidia-smi --query-gpu=index,uuid --format=csv,noheader'
if use_docker:
self.client.images.pull(cuda_image)
Expand Down
8 changes: 6 additions & 2 deletions codalab/worker/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -453,13 +453,17 @@ def parse_gpuset_args(arg):

try:
all_gpus = DockerRuntime().get_nvidia_devices() # Dict[GPU index: GPU UUID]
except DockerException:
except DockerException as e:
logger.error(e)
logger.error("Setting all_gpus to be empty...")
all_gpus = {}
# Docker socket can't be used
except requests.exceptions.ConnectionError:
try:
all_gpus = DockerRuntime().get_nvidia_devices(use_docker=False)
except SingularityError:
except SingularityError as e:
logger.error(e)
logger.error("Setting all_gpus to be empty...")
all_gpus = {}

if arg == 'ALL':
Expand Down