test_driver fails under openmpi compilation

[yixin-cfd]

yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/bin$ mpirun --version
mpirun (Open MPI) 4.1.6

Report bugs to http://www.open-mpi.org/community/help/
yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/bin$ mpic++ --version
g++ (Ubuntu 13.2.0-23ubuntu4) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/bin$ mpirun -np 1 ./test_driver_AD 
[JLX:390270] *** An error occurred in MPI_Win_create
[JLX:390270] *** reported by process [748552193,0]
[JLX:390270] *** on communicator MPI_COMM_WORLD
[JLX:390270] *** MPI_ERR_WIN: invalid window
[JLX:390270] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[JLX:390270] ***    and potentially your MPI job)
yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/bin$ mpirun -np 2  ./test_driver_AD 
===============================================================================
All tests passed (2 assertions in 1 test case)

===============================================================================
All tests passed (2 assertions in 1 test case)

yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/bin$ mpirun -np 1  ./test_driver
[JLX:391784] *** An error occurred in MPI_Win_create
[JLX:391784] *** reported by process [582418433,0]
[JLX:391784] *** on communicator MPI_COMM_WORLD
[JLX:391784] *** MPI_ERR_WIN: invalid window
[JLX:391784] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[JLX:391784] ***    and potentially your MPI job)
yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/bin$ mpirun -np 2  ./test_driver

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_driver is a Catch v2.13.9 host application.
Run with -? for options

-------------------------------------------------------------------------------
Geometry constructor
-------------------------------------------------------------------------------
../UnitTests/Common/geometry/CGeometry_test.cpp:33
...............................................................................

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_driver is a Catch v2.13.9 host application.
Run with -? for options

-------------------------------------------------------------------------------
Geometry constructor
-------------------------------------------------------------------------------
../UnitTests/Common/geometry/CGeometry_test.cpp:33

../UnitTests/Common/geometry/CGeometry_test.cpp:42: FAILED:
  CHECK( aux_geometry->GetnPoint() == 125 )
with expansion:
  62 == 125

../UnitTests/Common/geometry/CGeometry_test.cpp:43: FAILED:
  CHECK( aux_geometry->GetnElem() == 64 )
with expansion:
  42 == 64

../UnitTests/Common/geometry/CGeometry_test.cpp:44: FAILED:
  CHECK( aux_geometry->GetnElemHexa() == 64 )
with expansion:
  42 == 64

../UnitTests/Common/geometry/CGeometry_test.cpp:44: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGSEGV - Segmentation violation signal

===============================================================================
...............................................................................

../UnitTests/Common/geometry/CGeometry_test.cpp:42: FAILED:
  CHECK( aux_geometry->GetnPoint() == 125 )
with expansion:
  63 == 125

../UnitTests/Common/geometry/CGeometry_test.cpp:43: FAILED:
  CHECK( aux_geometry->GetnElem() == 64 )
with expansion:
  43 == 64

../UnitTests/Common/geometry/CGeometry_test.cpp:44: FAILED:
  CHECK( aux_geometry->GetnElemHexa() == 64 )
with expansion:
  43 == 64

test cases:  3 |  2 passed | 1 failed
assertions: 15 | 11 passed | 4 failed

[JLX:392626] *** Process received signal ***
[JLX:392626] Signal: Segmentation fault (11)
[JLX:392626] Signal code:  (-6)
[JLX:392626] Failing at address: 0x3e80005fdb2
[JLX:392626] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x74afaee45320]
[JLX:392626] [ 1] ./test_driver(+0x7240e)[0x55a72bb0d40e]
[JLX:392626] [ 2] ./test_driver(+0x9184a)[0x55a72bb2c84a]
[JLX:392626] [ 3] ./test_driver(+0xa52fd)[0x55a72bb402fd]
[JLX:392626] [ 4] ./test_driver(+0xb1f4a)[0x55a72bb4cf4a]
[JLX:392626] [ 5] ./test_driver(+0xbbe41)[0x55a72bb56e41]
[JLX:392626] [ 6] ./test_driver(+0xbc3ce)[0x55a72bb573ce]
[JLX:392626] [ 7] ./test_driver(+0x61c56)[0x55a72bafcc56]
[JLX:392626] [ 8] /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x74afaee2a1ca]
[JLX:392626] [ 9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x74afaee2a28b]
[JLX:392626] [10] ./test_driver(+0x69895)[0x55a72bb04895]
[JLX:392626] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
../UnitTests/Common/geometry/CGeometry_test.cpp:44: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGTERM - Termination request signal

===============================================================================
test cases:  3 |  2 passed | 1 failed
assertions: 17 | 13 passed | 4 failed

--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 0 on node JLX exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/bin$ 

mpi divided the nodes, can't test_driver pass in mpi environment?

[pcarruscag]

Probably not, the expected results are hardcoded for just one rank

[yixin-cfd]

One kernel does not run SU2_CFD, is it because of the latest ubuntu version 24.04 or openmpi version?

[TobiKattmann]

Can you please try this suggestion of using mpirun --mca osc ucx -np 1 ...

[yixin-cfd]

yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/bin$ mpirun --mca osc ucx -np 1 ./test_driver
[JLX:481640] *** An error occurred in MPI_Win_create
[JLX:481640] *** reported by process [2176122881,0]
[JLX:481640] *** on communicator MPI_COMM_WORLD
[JLX:481640] *** MPI_ERR_WIN: invalid window
[JLX:481640] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[JLX:481640] ***    and potentially your MPI job)
yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/bin$ mpirun --mca osc ucx -np 2 ./test_driver
[JLX:482351] *** An error occurred in MPI_Win_create
[JLX:482351] *** reported by process [2227568641,0]
[JLX:482351] *** on communicator MPI_COMM_WORLD
[JLX:482351] *** MPI_ERR_WIN: invalid window
[JLX:482351] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[JLX:482351] ***    and potentially your MPI job)
[JLX:482348] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[JLX:482348] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

[yixin-cfd]

yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/QuickStart$ mpirun --mca osc ucx -n 1 ./SU2_CFD ./inv_NACA0012.cfg 
[JLX:512796] *** An error occurred in MPI_Win_create
[JLX:512796] *** reported by process [200867841,0]
[JLX:512796] *** on communicator MPI_COMM_WORLD
[JLX:512796] *** MPI_ERR_WIN: invalid window
[JLX:512796] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[JLX:512796] ***    and potentially your MPI job)
yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/QuickStart$ mpirun -np 1 ./SU2_CFD ./inv_NACA0012.cfg 
[JLX:514863] *** An error occurred in MPI_Win_create
[JLX:514863] *** reported by process [62980097,0]
[JLX:514863] *** on communicator MPI_COMM_WORLD
[JLX:514863] *** MPI_ERR_WIN: invalid window
[JLX:514863] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[JLX:514863] ***    and potentially your MPI job)
yixin@JLX:/mnt/disk/linux/Code/Cpp/SU2/QuickStart$ mpirun -np 2 ./SU2_CFD ./inv_NACA0012.cfg 

-------------------------------------------------------------------------
|    ___ _   _ ___                                                      |
|   / __| | | |_  )   Release 8.0.1 "Harrier"                           |

[TobiKattmann]

What system setup are you using? Like: WSL, VM, native linux ... I am on WSL ubuntu 22.04 and that is working (with that hack) out of the box for me.

[yixin-cfd]

native Ubuntu 24.04.1 LTS

[TobiKattmann]

hmm, then I am out of ideas unfortunately