ubuntu 20.04 / slurm 21.08 / slurm_gpustat does not work as expected

[pankajd-57]

root@slurmsvr:~# slurm_gpustat --verbose

Under SLURM management

There are a total of 152 gpus [up]
152 NONAME_GPU gpus

There are a total of 128 gpus [accessible]
128 NONAME_GPU gpus

Usage by user:

There are 128 gpus available:
Traceback (most recent call last):
File "/usr/local/bin/slurm_gpustat", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/slurm_gpustat/slurm_gpustat.py", line 714, in main
all_info(color=args.color, verbose=args.verbose, partition=args.partition)
File "<@beartype(slurm_gpustat.slurm_gpustat.all_info) at 0x7fddcfcc6700>", line 69, in all_info
File "/usr/local/lib/python3.8/dist-packages/slurm_gpustat/slurm_gpustat.py", line 684, in all_info
available(resources=resources, states=states, verbose=verbose)
File "<@beartype(slurm_gpustat.slurm_gpustat.available) at 0x7fddcfcc65e0>", line 65, in available
File "/usr/local/lib/python3.8/dist-packages/slurm_gpustat/slurm_gpustat.py", line 650, in available
occupancy = occupancy_stats_for_node(node)
File "/usr/local/lib/python3.8/dist-packages/slurm_gpustat/slurm_gpustat.py", line 404, in occupancy_stats_for_node
cfg_val = metrics["CfgTRES"][metric]
KeyError: 'gres/gpu:a100-sxm4'

root@slurmsvr:~# slurm_gpustat

Under SLURM management

There are a total of 152 gpus [up]
152 NONAME_GPU gpus

There are a total of 128 gpus [accessible]
128 NONAME_GPU gpus

Usage by user:

There are 128 gpus available:
NONAME_GPU: 128 available