Advice on building with Intel Compiler and MPI

[sidiney-crescencio]

Hello everyone,

I'd like some advice/guidance on how to build SU2 from source using Intel Compiler and Intel MPI.

I'm using RHEL 9 and following pretty much these steps

This compiles successfully, but I start to get weird behaviors as job getting stuck when trying to run with more than 7 nodes.

I'm using SLURM, so I'd appreciate any help on this. I tried with newer versions but then I'm having different errors when compiling, so I'll wait for someone to reply first that maybe it help me in the right direction.

Thanks in advance!

git clone https://github.com/su2code/SU2.git
source /apps/oneapi2022.3.0/setvars.sh
export CC=mpiicc; export  CXX=mpiicpc ; export FC=mpiifort;
mkdir build; cd SU2
./meson.py build -Dwith-mpi=enabled -Denable-cgns=true -Denable-tecio=true -Denable-mkl=true --prefix=/home/cmsupport1/sidiney_experiment/build
export SU2_RUN=/home/cmsupport1/sidiney_experiment/build/bin
export SU2_HOME=/home/cmsupport1/sidiney_experiment/SU2
export PATH=$PATH:$SU2_RUN
export PYTHONPATH=$PYTHONPATH:$SU2_RUN
./ninja -C build install

[bigfooted]

Can you tell a bit more about the 'weird behavior' ? What exactly happens? Is it for every N>7? Does it happen all the time? And when N<7, do you see the expected speed-up with N?

[sidiney-crescencio]

sorry for the late reply @bigfooted , had some personal issues and was absent.

The iterations just stop from happening, so they get frozen somehow or stuck, linux processes are still on servers, but no progress and no apparent errors. Sometimes it's getting stuck even before starting the iterations

For instance

_------------------------------ Begin Solver -----------------------------

Simulation Run using the Single-zone Driver
WARNING: SU2 was not compiled for an AVX-capable architecture. Performance could be better,
see https://su2code.github.io/docs_v7/Build-SU2-Linux-MacOS/#compiler-optimizations_

Is it for every N>7 ?

Yes.

Does it happen all the time?

Yes.

And when N<7, do you see the expected speed-up with N?

It used to be a bit slower, since less MPI tasks.

But right now, I've changed a bit initial setup to see if would help

From RHEL 9 to RHEL 8 and trying with oneapi2024 , even for N<7 it's getting stuck

I tried to debug using gdb, and got the attached output gdb.txt , and tried to capture a tcpdump, but couldn't find anything pointing to something specific.

On gdb output, there seems to be a problem, but I don't know how to take it further.

[1]         domain=(MPIR_PMI_DOMAIN_LOCAL | MPIR_PMI_DOMAIN_NODE_ROOTS | unknown: 1020)) at ../../src/util/mpir_pmi.c:1352
[0]         domain=(MPIR_PMI_DOMAIN_LOCAL | MPIR_PMI_DOMAIN_NODE_ROOTS | unknown: 1020)) at ../../src/util/mpir_pmi.c:13
```52

[gdb.txt](https://github.com/su2code/SU2/files/13734619/gdb.txt)


[su2.txt](https://github.com/su2code/SU2/files/13734634/su2.txt)

If you have any ideas, I'd appreciate. Thanks.

[bigfooted]

maybe @jblueh has some suggestions for further debugging that can help pinpoint the problem. Do you have another mpi library like mpich available?