[YUNIKORN-2815] Run e2e tests against Kubernetes 1.31

[craigcondit]

What is this PR for?

Update the ee2e test matrix to include Kubernetes 1.31. Also, set the InPlacePodVerticalScaling feature flag when running on Kind v1.27.x or later to facilitate easier testing of pod resizing.

What type of PR is it?

• - Bug Fix
• - Improvement
• - Feature
• - Documentation
• - Hot Fix
• - Refactoring

Todos

• - Task

What is the Jira issue?

https://issues.apache.org/jira/browse/YUNIKORN-2815

How should this be tested?

Screenshots (if appropriate)

Questions:

• - The licenses files need update.
• - There is breaking changes for older versions.
• - It needs documentation.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.81%. Comparing base (2278b32) to head (31ae008).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #894      +/-   ##
==========================================
- Coverage   67.84%   67.81%   -0.03%     
==========================================
  Files          70       70              
  Lines        7485     7485              
==========================================
- Hits         5078     5076       -2     
- Misses       2195     2196       +1     
- Partials      212      213       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[craigcondit]

Test failures are all preemption - unrelated to this PR. All other e2e tests passed.

[wilfred-s]

VPA was pushed out late in the 1.31 release cycle. The list of fixes required to move to beta is big and some are really basic. There are a couple of really basic fixes that are only in 1.30 or 1.31.
I think we only ever want to test VPA with 1.31, maybe 1.30, and later. Before that too many things are broken.

[craigcondit]

I enabled the feature for 1.27 and later because that's when it was first implemented. The test we need to perform to validate that YuniKorn is behaving properly is simply to resize a pod and verify that the resource utilization within YuniKorn itself has been updated. That should be possible to test even as old as 1.27. Adding support within YuniKorn also doesn't affect older versions unless someone has actually turned the feature flag on.

[wilfred-s]

One of the basic problem described upstream is linked to testing. E2e testing of the feature has shown really bad performance when trying to use VPA. Changes taking the kubelet sync time, 60 seconds, to propagate and get actioned. Changes like these should always take a 1 second or so, but code is not working correctly.
Container spec defaulting was also changed between releases which means we might see fields in 1.31 set by default which we do not see in 1.27 or the other way around.

We just need to be careful that we're not adding tests that are flaky or can only work on the latest release etc.

[craigcondit]

Differences in various versions are precisely why we want to enable the flag for the widest possible cross-section of releases. Because users can turn this on, there's a possibility that they will. If we do find that there's a test that only works on newer versions, we can disable that test for older releases. However, we will not know this until we test on the widest possible range of releases.

[zhuqi-lucas]

LGTM