Make iptables initialization error non fatal #1497
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your PR. The following commands are available:
|
|
/test-all |
antoninbas
reviewed
Nov 5, 2020
| @@ -44,6 +45,10 @@ import ( | |||
| "github.com/vmware-tanzu/antrea/pkg/ovs/ovsconfig" | |||
| ) | |||
|
|
|||
| const ( | |||
| networkReadyTimeout = 30 * time.Second | |||
There was a problem hiding this comment.
what's the default kubelet / container runtime timeout for CNI Add? could you add the info as a comment?
There was a problem hiding this comment.
runtime request timeout is 2 minutes: https://github.com/kubernetes/kubernetes/blob/fc87c5927ca00eee2b437dc77b64f993728da87b/staging/src/k8s.io/kubelet/config/v1beta1/types.go#L451.
will add this information.
pkg/agent/route/route_linux.go
Outdated
| func (c *Client) initIPTablesOnce(done func()) { | ||
| defer done() | ||
| for { | ||
| time.Sleep(2 * time.Second) |
There was a problem hiding this comment.
curious about why you put a sleep before the first try?
There was a problem hiding this comment.
this was a mistake, thanks for catching it, now I understand why Initialize took more than 2 seconds in integration test even if I didn't make the lock hold..
|
/test-all |
|
/test-all |
Codecov Report
@@ Coverage Diff @@
## master #1497 +/- ##
==========================================
+ Coverage 68.16% 68.62% +0.46%
==========================================
Files 165 165
Lines 13107 13163 +56
==========================================
+ Hits 8934 9033 +99
+ Misses 3237 3193 -44
- Partials 936 937 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
|
antoninbas
previously approved these changes
Nov 6, 2020
|
/test-all |
In large scale clusters, xtables lock may be hold by kubelet/ kube-proxy/ portmap for a long time, especially when there are many service rules in nat table. antrea-agent may not be able to acquire the lock in short time. If the agent blocks on the lock or quit itself, the CNI server won't be running, causing all CNI requests to fail. If the Pods' restart policy is Always and there are dead Pods, container runtime will keep retrying calling CNIs, during which portmap is called first, leading to more xtables lock contention. This patch makes iptables initialization error non fatal and uses a goroutine to retry it until success. The agent will start the CNI server anyway and handle the CNI Del requests but won't handle CNI Add requests until the network is ready.
|
@antoninbas @jianjuns I rebased on master and fixed a lint error, could you approve again? /test-all |
|
/test-windows-conformance |
jianjuns
approved these changes
Nov 9, 2020
antoninbas
approved these changes
Nov 9, 2020
antoninbas
pushed a commit
to antoninbas/antrea
that referenced
this pull request
Nov 10, 2020
In large scale clusters, xtables lock may be hold by kubelet/ kube-proxy/ portmap for a long time, especially when there are many service rules in nat table. antrea-agent may not be able to acquire the lock in short time. If the agent blocks on the lock or quit itself, the CNI server won't be running, causing all CNI requests to fail. If the Pods' restart policy is Always and there are dead Pods, container runtime will keep retrying calling CNIs, during which portmap is called first, leading to more xtables lock contention. This patch makes iptables initialization error non fatal and uses a goroutine to retry it until success. The agent will start the CNI server anyway and handle the CNI Del requests but won't handle CNI Add requests until the network is ready.
antoninbas
pushed a commit
to antoninbas/antrea
that referenced
this pull request
Nov 10, 2020
In large scale clusters, xtables lock may be hold by kubelet/ kube-proxy/ portmap for a long time, especially when there are many service rules in nat table. antrea-agent may not be able to acquire the lock in short time. If the agent blocks on the lock or quit itself, the CNI server won't be running, causing all CNI requests to fail. If the Pods' restart policy is Always and there are dead Pods, container runtime will keep retrying calling CNIs, during which portmap is called first, leading to more xtables lock contention. This patch makes iptables initialization error non fatal and uses a goroutine to retry it until success. The agent will start the CNI server anyway and handle the CNI Del requests but won't handle CNI Add requests until the network is ready.
antoninbas
pushed a commit
that referenced
this pull request
Nov 11, 2020
In large scale clusters, xtables lock may be hold by kubelet/ kube-proxy/ portmap for a long time, especially when there are many service rules in nat table. antrea-agent may not be able to acquire the lock in short time. If the agent blocks on the lock or quit itself, the CNI server won't be running, causing all CNI requests to fail. If the Pods' restart policy is Always and there are dead Pods, container runtime will keep retrying calling CNIs, during which portmap is called first, leading to more xtables lock contention. This patch makes iptables initialization error non fatal and uses a goroutine to retry it until success. The agent will start the CNI server anyway and handle the CNI Del requests but won't handle CNI Add requests until the network is ready.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In large scale clusters, xtables lock may be hold by kubelet/
kube-proxy/ portmap for a long time, especially when there are many
service rules in nat table. antrea-agent may not be able to acquire the
lock in short time. If the agent blocks on the lock or quit itself, the
CNI server won't be running, causing all CNI requests to fail. If the
Pods' restart policy is Always and there are dead Pods, container
runtime will keep retrying calling CNIs, during which portmap is called
first, leading to more xtables lock contention.
This patch makes iptables initialization error non fatal and uses a
goroutine to retry it until success. The agent will start the CNI server
anyway and handle the CNI Del requests but won't handle CNI Add requests
until the network is ready.
Fixes #1499