Pods running on windows nodes cannot access the API server using the internal Kubernetes Service

[perithompson]

Describe the bug
Pods running on windows nodes cannot access the API server when trying to connect via the kubernetes internal service. It seems that the IP address is accessible from the pod but https traffic fails.

PS C:\> tnc kubernetes.default.svc.cluster.local -p 443
WARNING: TCP connect to (10.96.0.1 : 443) failed
ComputerName           : kubernetes.default.svc.cluster.local
RemoteAddress          : 10.96.0.1
RemotePort             : 443
InterfaceAlias         : vEthernet (iis-2019-6b0636)
SourceAddress          : 192.168.4.16
PingSucceeded          : True
PingReplyDetails (RTT) : 86 ms
TcpTestSucceeded       : False

This is not the case when accessing the API server via this service from the host.

To Reproduce

• Deploy Antrea cluster with windows node.
• Start a windows pod
• Attempt to connect to the kubernetes service via cluster IP or DNS name

Expected
API Server should be accessible on cluster IP service

Actual behavior
A clear and concise description of what's the actual behavior. If applicable, add screenshots, log messages, etc. to help explain the problem.

Versions:
Please provide the following information:

• Antrea version (Docker image tag). v0.12.0 (Containerd) from @ruicao93 but I think other versions are effected
• Kubernetes version: v1.19.1
• Container runtime: Containerd 1.4.3
• OVS: 2.14.0

Additional context

Tracing route to kubernetes.default.svc.cluster.local [10.96.0.1]
over a maximum of 30 hops:
  1    79 ms    74 ms    73 ms  192.168.4.1
  2    84 ms    77 ms    79 ms  kubernetes.default.svc.cluster.local [10.96.0.1]
Trace complete.
PS C:\> curl.exe -vvv https://kubernetes.default.svc.cluster.local
* Rebuilt URL to: https://kubernetes.default.svc.cluster.local/
*   Trying 10.96.0.1...
* TCP_NODELAY set
* connect to 10.96.0.1 port 443 failed: Timed out
* Failed to connect to kubernetes.default.svc.cluster.local port 443: Timed out
* Closing connection 0
curl: (7) Failed to connect to kubernetes.default.svc.cluster.local port 443: Timed out

E0119 02:51:23.483912    1128 proxysocket.go:208] I/O error: readfrom tcp 10.96.0.1:443->10.96.0.1:49752: read tcp 10.176.37.239:49753->10.176.37.186:6443: wsarecv: A con
nection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to 
respond.

.0.0,proto=0,tp_src=0,tp_dst=0),eth(src=68:4f:64:14:d8:9a,dst=01:00:0c:cc:cc:cd),in_port(2),eth_type(0x05ff)
2021-01-19T11:29:56.667Z|92859|dpif(handler11)|WARN|system@ovs-system: execute ct(zone=65520,nat),recirc(0x1) failed (not supported) on packet ip,vlan_tci=0x0000,dl_src=0
0:50:56:a6:97:d2,dl_dst=00:50:56:a6:fb:e3,nw_src=10.176.37.205,nw_dst=10.176.37.208,nw_proto=4,nw_tos=0,nw_ecn=0,nw_ttl=63
 with metadata skb_priority(0),skb_mark(0),in_port(2) mtu 0
2021-01-19T11:29:56.667Z|92860|dpif(handler11)|WARN|system@ovs-system: execute ct(zone=65520,nat),recirc(0x1) failed (not supported) on packet ip,vlan_tci=0x0000,dl_src=0
0:50:56:a6:fb:e3,dl_dst=00:50:56:a6:97:d2,nw_src=10.176.37.208,nw_dst=10.176.37.205,nw_proto=4,nw_tos=0,nw_ecn=0,nw_ttl=63
 with metadata skb_priority(0),skb_mark(0),in_port(2) mtu 0
2021-01-19T11:29:56.950Z|92861|dpif(handler11)|WARN|system@ovs-system: execute ct(zone=65520,nat),recirc(0x1) failed (not supported) on packet ip,vlan_tci=0x0000,dl_src=0
0:50:56:a6:95:2e,dl_dst=00:50:56:a6:83:25,nw_src=10.176.37.224,nw_dst=10.176.37.218,nw_proto=4,nw_tos=0,nw_ecn=0,nw_ttl=63
 with metadata skb_priority(0),skb_mark(0),in_port(2) mtu 0
2021-01-19T11:29:56.950Z|92862|dpif(handler11)|WARN|system@ovs-system: execute ct(zone=65520,nat),recirc(0x1) failed (not supported) on packet ip,vlan_tci=0x0000,dl_src=0
0:50:56:a6:95:2e,dl_dst=00:50:56:a6:83:25,nw_src=10.176.37.224,nw_dst=10.176.37.218,nw_proto=4,nw_tos=0,nw_ecn=0,nw_ttl=63
 with metadata skb_priority(0),skb_mark(0),in_port(2) mtu 0

[antoninbas]

That doesn't really help with this issue here :), but I just wanted to point out that the fact that the ping tests "succeeds" is a bit misleading here. ICMP traffic is not load-balanced to an actual Pod here. If I recall correctly, for Windows Nodes, we create a "Service interface" on the Node and all ClusterIPs are assigned to it. So when the ping test "succeeds" (or in the case of the traceroute), it's just the host replying directly to the ICMP requests, with no proxy / load-balancing involved.

[antoninbas]

@ruicao93 does it mean that all Pod-to-Service traffic is broken for Windows Nodes in the latest release, or is this some edge case?

[ruicao93]

@ruicao93 does it mean that all Pod-to-Service traffic is broken for Windows Nodes in the latest release, or is this some edge case?

@antoninbas : I think only the services like "kubernetes" are broken. It's because the endpointIP is node IP instead of pod IP.

[perithompson]

@antoninbas, @ruicao93 is correct, We've only seen this on the Kubernetes API internal service, I was going through this with him yesterday. The only thing that I could think of was that the service has a double hop readfrom tcp 10.96.0.1:443->10.96.0.1:49752: read tcp 10.176.37.239:49753->10.176.37.186:6443 although I don't know if that is unique? I wondered if this is the same result if you created a service similar to other external resources on an EndpointIP, but I haven't tested that yet.

[ruicao93]

@perithompson: The root cause should be that for such case, we both do DNAT and SNAT in same pipeline. Which is not supported by OVS as test.

So we need a new design to split these two NAT operation in different CT_ZONE. I will popose a new design soon.