[YUNIKORN-2808] E2E test Verify_preemption_on_priority_queue test is flaky

[manirajv06]

What is this PR for?

E2E test Verify_preemption_on_priority_queue test is flaky

What type of PR is it?

• - Bug Fix

Todos

• - Task

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

• - The licenses files need update.
• - There is breaking changes for older versions.
• - It needs documentation.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.89%. Comparing base (2278b32) to head (9c6a45d).
Report is 8 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #897      +/-   ##
==========================================
+ Coverage   67.84%   67.89%   +0.05%     
==========================================
  Files          70       70              
  Lines        7485     7488       +3     
==========================================
+ Hits         5078     5084       +6     
+ Misses       2195     2192       -3     
  Partials      212      212              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[manirajv06]

Preemption use single node for all these tests for simplicity. Failures happened only in Verify_preemption_on_priority_queue test. In this test, app submitted in low-priority queue has entered into reservation flow instead of going through preemption flow and eventually rejected because of its lower priority, followed by app waiting in high-priority queue not being able to complete the preemption because of node reserved by app submitted in low-priority queue. This specific situation doesn't happen always in all runs. Now, modified a test a bit to trigger the preemption while the headroom checks itself for both the apps so that high priority app is able to get resources through preemption whereas low priority app is not able to.

[craigcondit]

+1 LGTM.