OCPBUGS-18893: Rechecking pending Pods (conflict resolved)

[nicklesimba]

This fix resolves the issue where, after a forceful node reboot, force deleting a pod in a stateful set causes the pod to be recreated and remain indefinitely in the Pending state.

Solution description, as written on xagent003's upstream PR:

"We shouldn't treat all pending Pods as "alive" and skip the check. The list of all Pods fetch'd earlier may be stale, and as observed in some scenarios, several seconds before the ip-reconciler does the isPodAlive check.
Instead, can we retry a Get on an individual Pod, with the hopes that it has final IP/network annoations? So we try to refetch the pod a few times if it is Pending state and initial IP check fails. After that, just do the IP matching check like before"

Note that xagent003's upstream PR is stale and has since been rebased by dougbtv. You can find the current upstream PR here.

[openshift-ci-robot]

@nicklesimba: This pull request references Jira Issue OCPBUGS-18893, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.15.0) matches configured target version for branch (4.15.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This fix resolves the issue where, after a forceful node reboot, force deleting a pod in a stateful set causes the pod to be recreated and remain indefinitely in the Pending state.

Solution description, as written on xagent003's upstream PR:

"We shouldn't treat all pending Pods as "alive" and skip the check. The list of all Pods fetch'd earlier may be stale, and as observed in some scenarios, several seconds before the ip-reconciler does the isPodAlive check.
Instead, can we retry a Get on an individual Pod, with the hopes that it has final IP/network annoations? So we try to refetch the pod a few times if it is Pending state and initial IP check fails. After that, just do the IP matching check like before"

Note that xagent003's upstream PR is stale and has since been rebased by dougbtv. You can find the current upstream PR here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[maiqueb]

As spoken offline, this band-aid will cause stress on the API, thus impact workloads on other networks, including the cluster default network; the code should be refactored to rely on informers ASAP.

Having said that, this issue with pending pods is real, and is addressed by this PR.

@dougbtv we want to fix the pending pod issue, regardless, right ?

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: maiqueb, nicklesimba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [maiqueb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@nicklesimba: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-ci-robot]

@nicklesimba: Jira Issue OCPBUGS-18893: All pull requests linked via external trackers have merged:

• openshift/whereabouts-cni#196

Jira Issue OCPBUGS-18893 has been moved to the MODIFIED state.

In response to this:

This fix resolves the issue where, after a forceful node reboot, force deleting a pod in a stateful set causes the pod to be recreated and remain indefinitely in the Pending state.

Solution description, as written on xagent003's upstream PR:

"We shouldn't treat all pending Pods as "alive" and skip the check. The list of all Pods fetch'd earlier may be stale, and as observed in some scenarios, several seconds before the ip-reconciler does the isPodAlive check.
Instead, can we retry a Get on an individual Pod, with the hopes that it has final IP/network annoations? So we try to refetch the pod a few times if it is Pending state and initial IP check fails. After that, just do the IP matching check like before"

Note that xagent003's upstream PR is stale and has since been rebased by dougbtv. You can find the current upstream PR here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-merge-robot]

Fix included in accepted release 4.15.0-0.nightly-2023-09-27-073353

[openshift-merge-robot]

Fix included in accepted release 4.15.0-0.nightly-2024-01-13-050900