[YUNIKORN-2884] Task fail with post allocated but the pod will keep p…

[zhuqi-lucas]

…ending

What is this PR for?

Application should also release task fail case, pod should be deleted, because TerminationType_STOPPED_BY_RM will also be called when task fail.
If we don't delete the pod, the pod will be stuck in pending state.

After applied this PR, the test passed locally.

Note:
I will create a e2e testing jira after this PR resolved.

And in future, i am working on retry policy when task failing, we will ajust this logic to pod pending and deallocate in core side, but until now, we need to make sure the task failed should be cleaned for applications.

What type of PR is it?

• - Bug Fix
• - Improvement
• - Feature
• - Documentation
• - Hot Fix
• - Refactoring

Todos

• - Task

What is the Jira issue?

• Open an issue on Jira https://issues.apache.org/jira/browse//YUNIKORN-2884
• Put link here, and add [YUNIKORN-Jira number] in PR title, eg. [YUNIKORN-2] Gang scheduling interface parameters

How should this be tested?

Screenshots (if appropriate)

Questions:

• - The licenses files need update.
• - There is breaking changes for older versions.
• - It needs documentation.

[codecov]

Codecov Report

Attention: Patch coverage is 50.00000% with 5 lines in your changes missing coverage. Please review.

Project coverage is 68.15%. Comparing base (c959d23) to head (8b843cc).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/cache/application.go	37.50%	2 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #916      +/-   ##
==========================================
- Coverage   68.20%   68.15%   -0.05%     
==========================================
  Files          70       70              
  Lines        7621     7625       +4     
==========================================
- Hits         5198     5197       -1     
- Misses       2215     2218       +3     
- Partials      208      210       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[craigcondit]

I don't think this is the correct approach. We're arguably making things worse by terminating the pod, as this makes viewing events on it for diagnostic purposes near impossible. I think we should be looking into ways to recover from this issue instead.

[zhuqi-lucas]

Thanks @craigcondit,
I see, i will do the POC and design for the perfect solution.