Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OVN: The ovn loadbalancers (ClusterIP) seem to choke on jumboframes / icmp requesting frag to the service #1022

Closed
mangelajo opened this issue Dec 11, 2020 · 4 comments
Closed

OVN: The ovn loadbalancers (ClusterIP) seem to choke on jumboframes / icmp requesting frag to the service #1022

mangelajo opened this issue Dec 11, 2020 · 4 comments
Assignees
@mangelajo
Labels
blocker A blocker bug bug Something isn't working datapath Datapath related issues or enhancements OVN
Milestone
0.8.0

Comments

@mangelajo
Copy link
Contributor

What happened:

Our E2E tests fail since we test jumboframe fragmentation. (by chance... not by design)

https://mail.openvswitch.org/pipermail/ovs-discuss/2020-December/050834.html

iptables -I FORWARD -o ovn-k8s-gw0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1398

What you expected to happen:

Service tests to work

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Something that seems to workaround this, although it's not perfect, is to find the PMTU to the destination cluster....

sh-4.4# tracepath 10.133.2.2
 1?: [LOCALHOST]                      pmtu 8938
 1?: [LOCALHOST]                      pmtu 1438
 1:  ip-10-133-2-2.us-east-2.compute.internal              0.440ms reached
 1:  ip-10-133-2-2.us-east-2.compute.internal              0.410ms reached
     Resume: pmtu 1438 hops 1 back 1

And then clamp TCP/MSS with pmtu-40 (40 = IP + TCP overhead)

iptables -I FORWARD -o ovn-k8s-gw0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS  --set-mss 1398

This won't help with UDP, but will help until we can figure out a solution in ovn. (if it's really in ovn.)

This was tried without success:

iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS  --clamp-mss-to-pmtu

Environment:

  • Submariner version (use subctl version):
  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:
@mangelajo mangelajo added bug Something isn't working datapath Datapath related issues or enhancements OVN labels Dec 11, 2020
@mangelajo mangelajo added this to the 0.8.0 milestone Dec 11, 2020
@mangelajo mangelajo self-assigned this Dec 11, 2020
@mangelajo mangelajo added the blocker A blocker bug label Dec 11, 2020
@sridhargaddam
Copy link
Member

@mangelajo can you try if the following solution works with OVN?

#995 (comment)

@mangelajo
Copy link
Contributor Author

@mangelajo can you try if the following solution works with OVN?

#995 (comment)

I tried, it doesn't work in this case with OVN.

@mangelajo
Copy link
Contributor Author

I will be using this instead (note the -d) to be more selective on what's exactly clamped

iptables -I FORWARD -o ovn-k8s-gw0 -d 172.30.0.0/16 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS  --set-mss 1398

mangelajo added a commit to mangelajo/submariner that referenced this issue Dec 14, 2020
@mangelajo
Insert MSS clamping for traffic arriving services
95e415f 
OVN load balancing does not propagate fragment ICMPs down to the
services, so when fragmentation becomes necessary the TCP stack
of the service pod won't be able to detect this situation.

Until this is fixed in OVN this workaround (to be enhanced later)
will fix TCP.

Fixes-Issue: submariner-io#1022

Signed-off-by: Miguel Angel Ajo <[email protected]>
@mangelajo mangelajo mentioned this issue Dec 14, 2020
Merged
mangelajo added a commit to mangelajo/submariner that referenced this issue Dec 14, 2020
@mangelajo
Insert MSS clamping for traffic arriving services
22ce13a 
OVN load balancing does not propagate fragment ICMPs down to the
services, so when fragmentation becomes necessary the TCP stack
of the service pod won't be able to detect this situation.

Until this is fixed in OVN this workaround (to be enhanced later)
will fix TCP.

Fixes-Issue: submariner-io#1022

Signed-off-by: Miguel Angel Ajo <[email protected]>
mangelajo added a commit to mangelajo/submariner that referenced this issue Dec 14, 2020
@mangelajo
Insert MSS clamping for traffic arriving services
8661d7d 
OVN load balancing does not propagate fragment ICMPs down to the
services, so when fragmentation becomes necessary the TCP stack
of the service pod won't be able to detect this situation.

Until this is fixed in OVN this workaround (to be enhanced later)
will fix TCP.

Fixes-Issue: submariner-io#1022

Signed-off-by: Miguel Angel Ajo <[email protected]>
tpantelis pushed a commit that referenced this issue Dec 14, 2020
@mangelajo @tpantelis
Insert MSS clamping for traffic arriving services
05d50fc 
OVN load balancing does not propagate fragment ICMPs down to the
services, so when fragmentation becomes necessary the TCP stack
of the service pod won't be able to detect this situation.

Until this is fixed in OVN this workaround (to be enhanced later)
will fix TCP.

Fixes-Issue: #1022

Signed-off-by: Miguel Angel Ajo <[email protected]>
@mangelajo
Copy link
Contributor Author

The MSS clamping workaround is in place, so we can close this one for now, although we must keep an eye on OVN for a proper fix.

This was referenced Apr 21, 2021
Closed
Closed
@freedge freedge mentioned this issue Jun 22, 2023
Closed
@aswinsuryan aswinsuryan mentioned this issue Jul 22, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mangelajo mangelajo

Labels
blocker A blocker bug bug Something isn't working datapath Datapath related issues or enhancements OVN
Projects
None yet
Milestone
0.8.0
Development

No branches or pull requests
2 participants
@mangelajo @sridhargaddam