IPv6 support - including dual stack

[salv-orlando]

Describe what you are trying to solve
Antrea has currently been validate only with IPv4 networking.
IPv6 networking should be supported as well. This means both v6-only clusters and dual stack clusters. IPv6 dual stack has reached alpha status in Kubernetes 1.16

Describe the solution you have in mind

IPv6-only clusters

In theory IPv6 networking should work just like IPv4 networking, leveraging kube proxy for cluster IP implementation. There could however be some nuances of OVS flow syntax (e.g.: source and destination field names are different when using IPv6); there could also be some bugs pertaining IP address conversion and marshalling in Antrea components; regardless, no architectural change is expected

Dual stack clusters

This is more interesting as Antrea interfaces will need to be modified to support multiple pod IPs. Similarly the Antrea controller will now have to translate expressions for both IPv4 and IPv6 pod addresses.

On the agent side it might be worth considering a distinct table for IPv6 flows, and do that in a
way such only 1 table gets processed for each connection. This will probably be negligible from a performance perspective, but migth simplify implementation and troubleshooting.

Obviously Pod configuration and CNI plugin should take into account the presence of both a v4 and v6 address.

Additional we need to validate correct functionality of the Antrea agent with the kube-proxy in IPVS mode.

Describe how your solution impacts user flows
This should be fairly transparent to users. Obviously if the dual stack feature is enabled, workflows might be slightly different as each pod could have a v4 and v6 address, and services will select only endpoints from the specific address family for which they have been configured.

Describe the main design/architecture of your solution
No architectural change is foreseen on Antrea components.
The only design change would be a dedicated table for v6 ingress and egress network policies.
The current ones - the v4 tables - would be matched for v4 traffic only, and the new tables will be matches for v6 traffic only.

Alternative solutions that you considered
There isn't really an alternative solution, as this proposal is about extending support for features already available in Kubernetes

Test plan

1. Add end-2-end tests for IPv6 networking and dual stack networking. One idea could be to replicate existing tests in the three modes (v4, v6, dual-stack)
2. Add some dual-stack specific test. For sure we will need additional test cases for services

Additional context
Dual-stack IPv6 in Kubernetes: https://kubernetes.io/docs/concepts/services-networking/dual-stack/

[abhiraut]

can you link the google doc here? Thanks

[wenyingd]

The design doc is https://docs.google.com/document/d/1ufIwWeopZRaNKQNXhXx3sU89YloWztlk2oMMBzQdk28/edit#

[wenyingd]

I want to have some confirmation about the open issue mentioned in the design doc about if SNAT is supported with IPv6.

According to the discussion in the community meeting, a fully routed topology with IPv6 is strongly dependent on the infrastructure configuration on IPv6. So we could start with SNAT enabled in Encap mode as the first step, and then support No-SNAT in the NoEncap network.

Please share your thoughts, @salv-orlando @jianjuns @antoninbas @abhiraut @tnqn

[jianjuns]

According to the discussion in the community meeting, a fully routed topology with IPv6 is strongly dependent on the infrastructure configuration on IPv6. So we could start with SNAT enabled in Encap mode as the first step, and then support No-SNAT in the NoEncap network.

Sounds good to me!

[wenyingd]

According to the discussion, Encap mode should be supported at the first phase. Hence, following sub tasks might be needed

• Consume Node.Spec.CIDRs to parse multiple Pod Subnets [IPv6] Consume Node.Spec.CIDRs to support dual-stack configuration #971
• Improve IPAM plugin to support IPv6 [IPv6] Change host-local IPAM configuration for IPv6 #1039
• Add support for Pod L2 connectivity with IPv6 [IPv6] Change openflow pipeline for L2 Pod networking #1040
• Use separate fields for IPv4 and IPv6 instead of IP slice in configurations [IPv6] Use separate fields for IPv4 and IPv6 in GatewayConfig #1111
• Configure host routes and OpenFlow entries to support Pod L3 connectivity with IPv6 [IPv6] Implement L3 connectivity for IPv6 traffic #1011
• Add support for IPv6 in Service [IPv6] Add support for dual-stack when using kube-proxy for Service #1200
• Add support for matching IPv6 addresses in NetworkPolicy [IPv6] Add IPv6 support for NetworkPolicy by @wenyingd #1487
• Add support for using IPv6 in Traceflow [IPv6] Traceflow IPv6 support #1579
• Add enhancement in CI pipeline to support IPv6 test
• Enhance the user interfaces for IPv6, including writing/displaying IPv6 addresses in Octant and CRD [IPv6] Display dual stack NodeSubnet in antrea-octant-plugin #1156 [IPv6] Handle dual stack NodeSubnet for monitoring CRD #1182
• Enhance antctl to support Pod using IPv6 address, including OVS tracing and flow dumping [IPv6] Add support for IPv6 address in antctl and agent's apiserver #1118
• Add necessary changes in flow exporter to support IPv6 connections [IPv6] Support flow exporter #1541
• Support IPv6 in NoEncap mode
• Support IPv6 over IPv6 tunnel in Encap mode
• Verify IPv6 connections between Antrea Controller and Agent and made necessary changes

[antoninbas]

I notice that there is one remaining item for this issue:

Support IPv6 in NoEncap mode

Is this accurate (i.e. support is missing)? Does anyone know what's not supported and what needs to be done? Maybe opening a separate issue with more context would be a good idea.

[lzhecheng]

I am going to test it.

[tnqn]

@lzhecheng you may need to update GetNodeAddr to make dual-stack noEncap work. Please see the thread #2191 (comment)

[sdake]

General question here about IPv6 only support. Does Antrea support IPv6 only mode, and further, is the list accurate? I plan to test this, although it is easier said then done as photon does not yet have IPv6 support (so I have to roll with a different OS).

Cheers,
-steve

[antoninbas]

General question here about IPv6 only support. Does Antrea support IPv6 only mode, and further, is the list accurate? I plan to test this, although it is easier said then done as photon does not yet have IPv6 support (so I have to roll with a different OS).

Yes Antrea supports both IPv6-only and dual-stack clusters, and we test both configurations in CI. The list above should also be accurate. If anything, IPv6-only mode is easier to support, there are fewer quirks than with dual-stack and it's very similar to IPv4-only clusters.

[lzhecheng]

I just tested no-encap mode and found that there're failures in dual-stack e2e and IPv6-only conformance tests. For the first one, an issue has been created to track the bug #2426. The second will need more investigation.

[lzhecheng]

Used a PR to verify noencap support by all ipv6-only and dual-stack tests:

diff --git a/build/yamls/antrea.yml b/build/yamls/antrea.yml
index b75e88e6..a215daba 100644
--- a/build/yamls/antrea.yml
+++ b/build/yamls/antrea.yml
@@ -3693,7 +3693,7 @@ data:
     # networkPolicyOnly: Antrea enforces NetworkPolicy only, and utilizes CNI chaining and delegates Pod
     #                    IPAM and connectivity to the primary CNI.
     #
-    #trafficEncapMode: encap
+    trafficEncapMode: noEncap

     # Whether or not to SNAT (using the Node IP) the egress traffic from a Pod to the external network.
     # This option is for the noEncap traffic mode only, and
 the default value is false. In the noEncap

All tests are successful.