Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logic to handle multiple VPC CIDRs #234

Merged
merged 1 commit into from
Nov 20, 2018

Conversation

liwenwu-amazon
Copy link
Contributor

Issue #35 , if available:

Description of Changes:

  • modify IP rules for the Pod who is using non-primary ENI

    • if AWS_VPC_K8S_CNI_EXTERNALSNAT is set to false and packet's IP-DA is not in any VPC CIDRs range, use main routing table
    • Otherwise, use non-primary ENI's route table
  • modify IPTable rules

    • if AWS_VPC_K8S_CNI_EXTERNALSNAT is set to false , SNAT the packet with primary interface IP address if packet's IP-DA is not in any VPC CIDR ranges

Tests Performed

Testing Topology

  • Create VPC with at least 2 CIDRs (e.g. 10.0.0.0/16, 10.1.0.0/16)
  • at least 3 workers, node1, node2 in 10.0.0.0/16 and node3 in 10.1.0.0/16

Verify ping works and also verify expected ping traffic is going through expected ENI (using tcpdump) for following cases:

AWS_VPC_K8S_CNI_EXTERNALSNAT is set to false

	-  intra node1: 
		pod1 (eni-eth0) --> pod2 (eni-eth0) 
		pod1 (eni-eth0) --> pod3 (eni-eth1)
                pod3 (eni-eth1) ->  pod4 (eni-eth1)


	- inter node: (node1, node2 are in SAME CIDRs)
		node1, pod1 (eni-eth0) -> node2, pod1 (eni-eth0)
		node1, pod1 (eni-eth0) -> node2, pod3 (eni-eth1)
		node1, pod3 (eni-eth1) -> node2, pod3 (eni-eth1)

        - inter node: (node1, node3 are in 2 different CIDRs)
                node1, pod1 (eni-eth0) -> node3, pod1 (eni-eth0)
                node1, pod1 (eni-eth0) -> node3, pod3 (eni-eth1)
                node1, pod3 (eni-eth1) -> node3, pod3 (eni-eth1)

	- external node1:
		pod1(eni-eth0) --> www.yahoo.com
		pod3(eni-eth1) --> www.yahoo.com

	- extern node3
                pod1(eni-eth0) --> www.yahoo.com
                pod3(eni-eth1) --> www.yahoo.com

AWS_VPC_K8S_CNI_EXTERNALSNAT is set to true

        -  intra node1:
                pod1 (eni-eth0) --> pod2 (eni-eth0)          
                pod1 (eni-eth0) --> pod3 (eni-eth1)
                pod3 (eni-eth1) ->  pod4 (eni-eth1)


        - inter node: (node1, node2 are in SAME CIDRs)
                node1, pod1 (eni-eth0) -> node2, pod1 (eni-eth0)
                node1, pod1 (eni-eth0) -> node2, pod3 (eni-eth1)
                node1, pod3 (eni-eth1) -> node2, pod3 (eni-eth1)

        - inter node: (node1, node3 are in 2 different CIDRs)
                node1, pod1 (eni-eth0) -> node3, pod1 (eni-eth0)
                node1, pod1 (eni-eth0) -> node3, pod3 (eni-eth1)
                node1, pod3 (eni-eth1) -> node3, pod3 (eni-eth1)

        - external node1:
                pod1(eni-eth0) --> www.yahoo.com
                pod3(eni-eth1) --> www.yahoo.coma // verify ping are going out eth1

        - extern node3
                pod1(eni-eth0) --> www.yahoo.com
                pod3(eni-eth1) --> www.yahoo.coa // verify ping are going out eth1

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Copy link
Contributor

@mogren mogren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will take another look in the morning, thanks for adding this!

// NAT gateway rather than on node. Failure to parse the setting will result in a log and the
// setting will be disabled.
func (n *linuxNetwork) UseExternalSNAT() bool {
return getBoolEnvVar(envExternalSNAT, false)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. Could it just be return useExternalSNAT() instead, since they do the same thing? Or should the env variable be saved as a property on the linuxNetwork?

pkg/networkutils/network.go Outdated Show resolved Hide resolved
pkg/networkutils/network.go Outdated Show resolved Hide resolved
log.Errorf("Failed to add fromContainer rule for %s err: %v", addr.String(), err)
return errors.Wrap(err, "add NS network: failed to add fromContainer rule")
if useExternalSNAT {
// add rule: 1536: from <podIP> use table <table>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the 1536: mean?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the priority of the ip rule.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

}
log.Infof("Added rule priority %d from %s table %d", fromContainerRulePriority, addr.String(), table)
} else {
// add rule: 1536: list of from <podIP> to <vpcCIDR> use table <table>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, why the number?

plugins/routed-eni/driver/driver.go Outdated Show resolved Hide resolved
return nil
}

if toFlag {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You pass in !c.networkClient.UseExternalSNAT() to set this boolean. I think it needs a more descriptive name, maybe something as straight forward as useExternalSNAT (and then flip the if-statements of course.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to keep useExternalSNAT on level above UpdateRuleListBySrc() call.

ipamd/rpc_handler.go Show resolved Hide resolved
{
name: fmt.Sprintf("rule for primary address %s", primaryAddr),
// build IPTABLES chain for SNAT of non-VPC outbound traffic
for i, _ := range vpcCIDRs {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this for-loop was from 0 to len(vpcCIDRs), there would be no need for the duplicate code on lines 256-262:

	for i := 0; i <= len(vpcCIDRs); i++ {

Copy link
Contributor

@mogren mogren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still some refactors that would improve the readability, but they can wait a bit.

ipamd/rpc_handler.go Show resolved Hide resolved
@liwenwu-amazon liwenwu-amazon merged commit 3fba7b6 into aws:master Nov 20, 2018
mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request Nov 27, 2018
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
@gflarity
Copy link

gflarity commented Dec 5, 2018

Awesome, I believe we ran into this... how do we get/run the latest version with this fix on EKS? Right now we're disabling SNAT with the environment variable.

Edit: Oh, looks like this the fix that uses the env var?

mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request Dec 20, 2018
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
@mayfield
Copy link

mayfield commented Jan 1, 2019

Was losing my mind trying to understand why my EKS networking was totally unreliable and this was my issue. Thank you so much @liwenwu-amazon!

mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request Jan 9, 2019
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request Jan 14, 2019
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request Feb 1, 2019
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request Feb 3, 2019
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request Feb 7, 2019
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request Feb 27, 2019
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request Mar 7, 2019
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
mogren added a commit that referenced this pull request Mar 12, 2019
rudoi pushed a commit to newrelic-forks/amazon-vpc-cni-k8s that referenced this pull request Apr 1, 2019
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request Apr 25, 2019
Two small bugfixes in the error handling when deleting rules,
otherwise no functional changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants