rt.sh gun pdlib jobs hanging

[jiandewang]

I am testing new MOM6 code on HERA but somehow the two gnu pdlib jobs are hanging:
/opal/mca/btl/tcp/btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(414) failed: Connection reset by peer

you can see my err file at /scratch1/NCEPDEV/stmp2/Jiande.Wang/FV3_RT/rt_3359678/cpld_debug_pdlib_p8_gnu/err

has anyone encountered this kind of situation before ? is it a system or library issue ?
all other jobs ran fine

[jiandewang]

the issue could be related to my code as it didn't appear in develop branch after I tried

[DeniseWorthen]

@jiandewang On Hera, I got a similar error for this test using the current develop of MOM6 + CMEPS PR. So it is not your MOM6 changes. When I re-ran in the same run directory it ran fine. You might want to try just re-running in the same failed directory.

[jiandewang]

@DeniseWorthen thanks for the information. I only tried develop branch once and that didn't give me error. For MOM6 PR code I am testing the issue is reproduciable as I tried at laest 6 times at different times on HERA but the issue doesn't exist on hercules.
However on HERA if I only 20 CPU on each node, then job runs fine. Also if I increase OMP_STACKSIZE to 800M, job runs fine.

this is the error message from log file:
_WARNING: Open MPI failed to TCP connect to a peer MPI process. This
should not happen.

Your Open MPI job may now hang or fail.

Local host: h11c27
PID: 3783435
Message: connect() to 10.184.11.22:1032 failed
Error: Resource temporarily unavailable (11)

For the MOM6 PR, if I remove one of the "KPP commit" which contains some new 3D variable initialization, then the problem goes away. So the problem is realted to insufficient memory.

[jiandewang]

@junwang-noaa @DeniseWorthen @sanAkel
in UWM development branch we are using 20PE for ocean for this job. I increased to 28PE and the job running fine (and reproduciable) now. Increase it to 24 still got one failed case (out of my 5 tries). Do you think this is sufficient evidence to give greenlight to GFDL ?

[sanAkel]

@jiandewang Thanks for adding me to this thread.

I need to familiarize myself with the issue...

But as far as I understand there is a stack/memory issue, which goes away - so that sounds good!

I did have a chat with @marshallward yesterday and he said maybe he will get rid of some lines which now introduce 3d fields. He'll be able to comment on that.

I have nothing else to say at this moment. 🙏

[DeniseWorthen]

@jiandewang So 28PE resources for OCN in this test reliably works, right?

[jiandewang]

@DeniseWorthen I tried 20 times with 28PE and all run fine.

[DeniseWorthen]

And how did you implement the change? Did you just "bump" the OCN resources in the test?

[jiandewang]

add OCN_tasks="$((OCN_tasks_cpl_unstr + 8))" in test/tests/cpld_debug_pdlib_p8

[DeniseWorthen]

OK, this is what we did for the pdlib wave tasks also for the debug test. That isolates the extra resources to just that test, so I think this is fine to use.

[marshallward]

I had a look at these variables, and although they appear in a few OpenMP blocks, they are all declared as shared(). So I am unsure how they could have used OMP stack memory.

Initializing them could well increase the memory usage; not every compiler is going to store them in BSS, particularly a GNU debug build. It is the OpenMP stack memory that is confusing me. But maybe, as suggested by others, this is just a symptom of high memory usage and too many things bumping up against each other.

It might take a serious memory inspection to figure out what's going on.