Add --random-fully to SNAT IP Table Rules

[Scoobed]

Describe the problem/challenge you have
Default SNAT behavior is to preserve the source whenever possible. We are seeing a precentage of traffic using SNATs traffic fail due to TCP Port Numbers being reused too quickly. As the referenced in a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker below where there is latency in the insert into the conntrack table which can cause the same Source Port to be given to multiple pods.

Thus we are seeing in the TCP Dump output, there are multiple [TCP Port Numbers reused] on POD.

Describe the solution you'd like
Expose an option to randomize the Source Pods Ports to use --random / --random-fully in iptables

Anything else you would like to add?

See the Netfilter NAT & Conntrack kernel modules section on the following link.
Note the following link is Kubernetes / Docker, but looks like it would hold true with Kubernetes/Containerd level

a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker
time-wait article
Cilium SNAT issue

[antoninbas]

I personally support adding this as a configuration parameter for the antrea-agent, as I don't think there is any drawback (we would keep the current behavior as the default) and it could potentially help in some situations.

@Scoobed for your specific issue, there may be better ways to address it on the client side (we discussed this on Slack).
Regarding the links you shared: if memory served me right, the primary concern was race conditions in iptables / netfilter which impacted UDP specifically and therefore impacted DNS queries in K8s clusters. I am not sure TCP connections were affected. See https://quentin.machu.dev/2018/06/24/5-15s-dns-lookups-on-kubernetes/. In your case (based on our Slack conversation), there is no race condition as the implementation behaves correctly, but some network device in the path is not happy about rapid port reuse.

By the way, this may be a "good first issue" if you (or someone else) are interested in taking a stab at it. The only thing to look out for AFAIK is that some older kernel versions may not support --random-fully and that a check may be desired in the antrea-agent when the configuration parameter is set.

[antoninbas]

While this is not mentioned explicitly in the original post, I believe that @Scoobed experienced the issue in the context of Egress SNAT, given that the Slack post mentioned "SNAT IP per namespace".

However, the ability for users to configure --random-fully could apply both to Egress SNAT and to Node SNAT.

Node SNAT (masquerade):

antrea/pkg/agent/route/route_linux.go

Lines 979 to 987 in e921a6e

	if !c.noSNAT {
	writeLine(iptablesData, []string{
	"-A", antreaPostRoutingChain,
	"-m", "comment", "--comment", `"Antrea: masquerade Pod to external packets"`,
	"-s", podCIDR.String(), "-m", "set", "!", "--match-set", podIPSet, "dst",
	"!", "-o", c.nodeConfig.GatewayConfig.Name,
	"-j", iptables.MasqueradeTarget,
	}...)
	}

Egress SNAT:

antrea/pkg/agent/route/route_linux.go

Lines 969 to 978 in e921a6e

	for snatMark, snatIP := range snatMarkToIP {
	// Cannot reuse snatRuleSpec to generate the rule as it doesn't have "`" in the comment.
	writeLine(iptablesData, []string{
	"-A", antreaPostRoutingChain,
	"-m", "comment", "--comment", `"Antrea: SNAT Pod to external packets"`,
	"!", "-o", c.nodeConfig.GatewayConfig.Name,
	"-m", "mark", "--mark", fmt.Sprintf("%#08x/%#08x", snatMark, types.SNATIPMarkMask),
	"-j", iptables.SNATTarget, "--to", snatIP.String(),
	}...)
	}

It is not completely clear to me whether we would want 2 separate configuration parameters or a unified one. I would personally prefer the latter.

[Scoobed]

I would prefer two settings if possible as we might want the one but not the other one. if I was going to do the pull request is there directions on how this tested and thos requirements

[antoninbas]

I would prefer two settings if possible as we might want the one but not the other one

Do you mean that you would want source port selection to be random for Egress SNAT but not for Node SNAT (or the reverse)? Could you provide more information as to why?

[Scoobed]

I mean I would just like to control them independent of each other. Like only enable with Egress and not the node. Sorry if that came off confusing as I don't have a good reason other than flexibility.