Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CPDEV-100496] Ensure to use kubectl after wait for first control plane pods #646

Merged
merged 1 commit into from
May 2, 2024
Merged

[CPDEV-100496] Ensure to use kubectl after wait for first control plane pods #646

merged 1 commit into from
May 2, 2024

Conversation

ilia1243
Copy link
Contributor

Description

  • After [CPDEV-99413] Patch kubelet-config ConfigMap during upgrade #627 upgrade sometimes fail during call to components.patch_kubelet_configmap. Reproduced on rhel 9.2, upgrade v1.26.11 -> v1.27.8, full HA.
  • The reason is because kubeadm apply updates kubelet-config ConfigMap using not yet upgraded node with working control plane components, while components.patch_kubelet_configmap may randomly go through node with not yet working kube-apiserver.

Solution

  • Move components.patch_kubelet_configmap after waiting for readiness of control plane pods.

Test Cases

TestCase 1

Test Configuration:

  • OS: rhel 9.2
  • Inventory: Kubernetes v1.26.11, 1 balancer, 3 control-plane & worker nodes.

Steps:

  1. Install the cluster
  2. Upgrade to v1.27.8

Results:

Before After
Upgrade sometimes fail on kubernetes task Upgrade is successful.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • Integration CI passed
  • Unit tests. If Yes list of new/changed tests with brief description
  • There is no merge conflicts

@ilia1243 ilia1243 added the bug Something isn't working label Apr 23, 2024
@ilia1243 ilia1243 requested a review from Imadzuma April 23, 2024 06:35
@ilia1243 ilia1243 force-pushed the bugfix/upgrade_kubectl_after_wait branch from 12c2120 to 54682f3 Compare April 23, 2024 06:37
@koryaga koryaga self-requested a review April 26, 2024 09:38
@ilia1243 ilia1243 force-pushed the bugfix/upgrade_kubectl_after_wait branch from 54682f3 to 4436dbc Compare April 26, 2024 10:01
@andrewluckyguy
Copy link
Collaborator

PR have been verified with some issues

@OleksandrIaresko OleksandrIaresko self-requested a review May 1, 2024 06:55
@koryaga koryaga merged commit d0600d0 into main May 2, 2024
43 checks passed
@koryaga koryaga deleted the bugfix/upgrade_kubectl_after_wait branch May 2, 2024 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@koryaga koryaga koryaga approved these changes

@OleksandrIaresko OleksandrIaresko OleksandrIaresko approved these changes

@Imadzuma Imadzuma Awaiting requested review from Imadzuma

Assignees

@koryaga koryaga

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ilia1243 @andrewluckyguy @koryaga @OleksandrIaresko