ci: GH200 test instability

[edopao]

It was observed that the CI tests on GPU randomly hang, and eventually time out. In an attempt to make the CI stable on GH200 nodes, this PR proposes 2 changes:

• Reduce pytest parallelism, by lowering the num processes from 32 to 16 (note that GPU tests exploit CUDA MPS)
• Reduce the SLURM timeout, in order to early detect when jobs hang (longest job takes 10 min)

Additional change:

• Improve the hash on the Dockerfile for caching of base image: now also include the build arguments

[edopao]

I need to check with @finkandreas if CUDA MPS on GH200 is configured with multi-GPU support, because that would be desirable in our configuration: https://docs.nvidia.com/deploy/mps/#mps-on-multi-gpu-systems

[edopao]

I need to check with @finkandreas if CUDA MPS on GH200 is configured with multi-GPU support, because that would be desirable in our configuration: https://docs.nvidia.com/deploy/mps/#mps-on-multi-gpu-systems

@finkandreas Please ignore this question, No, it is not relevant to us. From the docs:

When CUDA_VISIBLE_DEVICES is set before launching the control daemon, the devices will be remapped by the MPS server.

I was hoping that MPS could automatically dispatch the CUDA kernels across different devices, but this is not the case.

[havogt]

Thanks!