After the Jobs are stopped, Job PodTemplate Labels are not restored only during unit testing

[tenzen-y]

What happened:
Only during unit testing, the Job PodTemplate Labels (.spec.template.labels) are not restored.
But, during integration testing and actual cluster running, they are restored after the Jobs are suspended.

What you expected to happen:
The Job PodTemplate Labels are always restored after the Jobs are stopped.

How to reproduce it (as minimally and precisely as possible):
We can see the reproduced PR and CI logs there:

• DO NOT MERGE: TESTING #3081
• https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/3081/pull-kueue-test-unit-main/1836114474063368192

Anything else we need to know?:
We verify if those labels (.spec.template.labels) are restored only during the integration testing, and the testing always successful:

kueue/test/integration/controller/jobs/job/job_controller_test.go

Lines 756 to 767 in e812a06

	ginkgo.By("verify the PodSetUpdates are restored", func() {
	// In case of batch/Job the stop is done with multiple API calls, suspended=true being
	// done before the info restoration. We should retry the read if the Info is not restored.
	gomega.Eventually(func(g gomega.Gomega) {
	g.Expect(k8sClient.Get(ctx, *jobLookupKey, createdJob)).Should(gomega.Succeed())
	g.Expect(createdJob.Spec.Template.Annotations).ShouldNot(gomega.HaveKey("ann1"))
	g.Expect(createdJob.Spec.Template.Annotations).Should(gomega.HaveKeyWithValue("old-ann-key", "old-ann-value"))
	g.Expect(createdJob.Spec.Template.Labels).ShouldNot(gomega.HaveKey("label1"))
	g.Expect(createdJob.Spec.Template.Labels).Should(gomega.HaveKeyWithValue("old-label-key", "old-label-value"))
	g.Expect(createdJob.Spec.Template.Spec.NodeSelector).ShouldNot(gomega.HaveKey(instanceKey))
	g.Expect(createdJob.Spec.Template.Spec.NodeSelector).ShouldNot(gomega.HaveKey("selector1"))
	}, util.Timeout, util.Interval).Should(gomega.Succeed())

Environment:

• Kubernetes version (use kubectl version):
• Kueue version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[mszadkow]

/assign

[mszadkow]

@tenzen-y please have a look at my explanation here: https://github.com/kubernetes-sigs/kueue/pull/3081/files#r1766613892

[tenzen-y]

@tenzen-y please have a look at my explanation here: https://github.com/kubernetes-sigs/kueue/pull/3081/files#r1766613892

As I commented the above PR, this is my misunderstanding.
@mszadkow Thank you for the investigation, again!
/close

[k8s-ci-robot]

@tenzen-y: Closing this issue.

In response to this:

@tenzen-y please have a look at my explanation here: https://github.com/kubernetes-sigs/kueue/pull/3081/files#r1766613892

As I commented the above PR, this is my misunderstanding.
@mszadkow Thank you for the investigation, again!
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tenzen-y]

/remove-kind bug
/kind support