Support using a LoadBalancer service in front of the submariner-gateway

[mangelajo]

What would you like to be added:

Automatic LoadBalancer creation that points to each gateway node, providing an external IP.

Why is this needed:

Currently the gateway nodes need to have a public IP address that can be targeted by other nodes,
but that's not supported on some deployment models, and something that we workaround via terraform
scripts.

Items:

• Pass down pod name into submariner-gateway submariner-operator#1389 submariner-operator to pass down the pod name into the gateway pod, so it can self-label when active.
• Make submariner-gateway label it's own Pod as gateway=active and host=$host when active #1370 Make submariner-gateway label it's own Pod as gateway=active and host=$host when active
• Allow subctl/operator --loadbalancer mode submariner-operator#1387 Add a setting to the operator and subctl to enable load-balancer mode. Once enabled creates the LoadBalancer automatically.
• Accept default public IP setting from environment variable #1389 Submariner to accept public-ip configuration via environment-variable from the operator.
• Support no-dedicated gateway nodes as part of the LoadBalancer support effort. cloud-prepare#49 Update cloud-prepare to allow no dedicated gateways.
• LB draft submariner-website#525 Document the LoadBalancer usage.
• Make the PublicIP field of Endpoints compatible with DNS entries #1369 Make the PublicIP field of Endpoints compatible with DNS entries
• Allow Deployment vs Daemonset model based on submariner.io/gateway=true labels in the context of loadbalancer mode (we need a solution for co-located services and anti-affinity)
  Depends on Verify if ipsec-preferred server will work with a network LoadBalancer on AWS #1310

[raffaelespazzoli]

this may help with the asymmetry between inbound and outbound IP:
https://tools.ietf.org/html/rfc3715

[mangelajo]

@raffaelespazzoli Is the LoadBalancer resource available on all clouds? I guess yes, but I had a question about that.

[raffaelespazzoli]

This questions is a bit confusing. LoadBalancer is not a resource, it's a type of service and the Service is a core resource available in Kubernetes. Is the LoadBalacer type of service honored in all of the clouds? It depends what you include in that definition of cloud. For the big commercial clouds the answer is yes.

[nyechiel]

This need further exploration work:

1. Basic research to determine which Kubernetes distros and OpenShift infrastructure providers actually support services of type LoadBalacer and how they work
2. Can we make this work with Submariner's IPsec (UDP) connections? If yes, how exactly does it simplify the user experience/preparation steps
3. Impact on performance vs. the current (direct) tunnelling approach
4. How HA would work
5. Other pros/cons vs. the current tunnelling approach

[raffaelespazzoli]

@nyechiel on point #1, the research is not so much about the kube distro, but about the cloud provider and how it a loadbalancer is implemented with the underlying cloud infrastructure.

[nyechiel]

@nyechiel on point #1, the research is not so much about the kube distro, but about the cloud provider and how it a loadbalancer is implemented with the underlying cloud infrastructure.

By "distros" I meant the managed Kubernetes offerings (AKS, EKS, GKE, etc) which are very popular among our users.

[mangelajo]

We tried the following model

One network LoadBalancer for IPSEC is created on each cluster for each gateway, and that LoadBalancer is announced
as the public endpoint of the cluster to establish IPSEC connections.

At this point the connections (at least in the way we configure them via whack) don't establish. Pluto
doesn't seem to identify the arriving packets properly (as outgoing traffic using the SNAT GATEWAY ip)
and not the LoadBalancer IP, for example:

When GW1 tries to contact GW2 LB on 3.139.173.17, it will leave the left private network using the SNAT
gateway IP 3.23.33.12, arrive 3.139.173.17, and then be directed to 10.0.130.185. GW2 will be confused
because it knows nothing about the SNAT gateway IP 3.23.33.12.

Same in the other direction. Sometimes it gets even weirder (you will see in the attached logs that on
gw1 the traffic seems to come from 169.254.1.243, that's just an artifact of how the network plugin of
that cluster works, which will not preserve the source IP of the connection.

pluto always end up responding with NO_PROPOSAL_CHOSEN

Feb 18 17:02:41.038572: | ISAKMP_v2_IKE_SA_INIT message received on 10.0.132.131:4500 but no connection has been authorized with policy ........
Feb 18 17:02:41.038579: packet from 169.254.1.243:55084: ISAKMP_v2_IKE_SA_INIT message received on 10.0.132.131:4500 but no suitable connection found with IKEv2 policy
Feb 18 17:02:41.038585: packet from 169.254.1.243:55084: responding to IKE_SA_INIT (34) message (Message ID 0) with unencrypted notification NO_PROPOSAL_CHOSEN

I was expecting that pluto would at least identify the packets received based on the ID that we configure (the internal IP of the
host).

WARNING: for the following details the IP addresses don't fully match the above diagrams because they belong
to different runs. I can update the diagrams if it's too confusing.

For example, on the left cluster we configure the connections with whack:

$ grep "Executing whack" side_a
I0218 17:02:40.718802       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-b-02-18-21-1613652198-10-0-130-185-0-0 --id 10.0.132.131 --host 10.0.132.131 --client 10.0.132.131/32 --ikeport 4500 --to --id 10.0.130.185 --host 3.139.173.17 --client 10.0.130.185/32 --ikeport 4500]
I0218 17:02:40.784762       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-b-02-18-21-1613652198-10-0-130-185-0-1 --id 10.0.132.131 --host 10.0.132.131 --client 10.0.132.131/32 --ikeport 4500 --to --id 10.0.130.185 --host 3.139.173.17 --client 172.31.0.0/16 --ikeport 4500]
I0218 17:02:40.829639       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-b-02-18-21-1613652198-10-0-130-185-0-2 --id 10.0.132.131 --host 10.0.132.131 --client 10.0.132.131/32 --ikeport 4500 --to --id 10.0.130.185 --host 3.139.173.17 --client 10.132.0.0/14 --ikeport 4500]
I0218 17:02:40.865794       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-b-02-18-21-1613652198-10-0-130-185-1-0 --id 10.0.132.131 --host 10.0.132.131 --client 172.30.0.0/16 --ikeport 4500 --to --id 10.0.130.185 --host 3.139.173.17 --client 10.0.130.185/32 --ikeport 4500]
I0218 17:02:40.869949       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-b-02-18-21-1613652198-10-0-130-185-1-1 --id 10.0.132.131 --host 10.0.132.131 --client 172.30.0.0/16 --ikeport 4500 --to --id 10.0.130.185 --host 3.139.173.17 --client 172.31.0.0/16 --ikeport 4500]
I0218 17:02:40.873664       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-b-02-18-21-1613652198-10-0-130-185-1-2 --id 10.0.132.131 --host 10.0.132.131 --client 172.30.0.0/16 --ikeport 4500 --to --id 10.0.130.185 --host 3.139.173.17 --client 10.132.0.0/14 --ikeport 4500]
I0218 17:02:40.882140       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-b-02-18-21-1613652198-10-0-130-185-2-0 --id 10.0.132.131 --host 10.0.132.131 --client 10.128.0.0/14 --ikeport 4500 --to --id 10.0.130.185 --host 3.139.173.17 --client 10.0.130.185/32 --ikeport 4500]
I0218 17:02:40.885935       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-b-02-18-21-1613652198-10-0-130-185-2-1 --id 10.0.132.131 --host 10.0.132.131 --client 10.128.0.0/14 --ikeport 4500 --to --id 10.0.130.185 --host 3.139.173.17 --client 172.31.0.0/16 --ikeport 4500]
I0218 17:02:40.896272       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-b-02-18-21-1613652198-10-0-130-185-2-2 --id 10.0.132.131 --host 10.0.132.131 --client 10.128.0.0/14 --ikeport 4500 --to --id 10.0.130.185 --host 3.139.173.17 --client 10.132.0.0/14 --ikeport 4500]

And on the right cluster:

I0218 17:02:36.913210       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-a-02-18-21-1613652198-10-0-132-131-0-0 --id 10.0.130.185 --host 10.0.130.185 --client 10.0.130.185/32 --ikeport 4500 --to --id 10.0.132.131 --host 3.23.239.128 --client 10.0.132.131/32 --ikeport 4500]
I0218 17:02:37.041705       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-a-02-18-21-1613652198-10-0-132-131-0-1 --id 10.0.130.185 --host 10.0.130.185 --client 10.0.130.185/32 --ikeport 4500 --to --id 10.0.132.131 --host 3.23.239.128 --client 172.30.0.0/16 --ikeport 4500]
I0218 17:02:37.120782       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-a-02-18-21-1613652198-10-0-132-131-0-2 --id 10.0.130.185 --host 10.0.130.185 --client 10.0.130.185/32 --ikeport 4500 --to --id 10.0.132.131 --host 3.23.239.128 --client 10.128.0.0/14 --ikeport 4500]
I0218 17:02:37.192190       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-a-02-18-21-1613652198-10-0-132-131-1-0 --id 10.0.130.185 --host 10.0.130.185 --client 172.31.0.0/16 --ikeport 4500 --to --id 10.0.132.131 --host 3.23.239.128 --client 10.0.132.131/32 --ikeport 4500]
I0218 17:02:37.196776       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-a-02-18-21-1613652198-10-0-132-131-1-1 --id 10.0.130.185 --host 10.0.130.185 --client 172.31.0.0/16 --ikeport 4500 --to --id 10.0.132.131 --host 3.23.239.128 --client 172.30.0.0/16 --ikeport 4500]
I0218 17:02:37.200965       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-a-02-18-21-1613652198-10-0-132-131-1-2 --id 10.0.130.185 --host 10.0.130.185 --client 172.31.0.0/16 --ikeport 4500 --to --id 10.0.132.131 --host 3.23.239.128 --client 10.128.0.0/14 --ikeport 4500]
I0218 17:02:37.205025       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-a-02-18-21-1613652198-10-0-132-131-2-0 --id 10.0.130.185 --host 10.0.130.185 --client 10.132.0.0/14 --ikeport 4500 --to --id 10.0.132.131 --host 3.23.239.128 --client 10.0.132.131/32 --ikeport 4500]
I0218 17:02:37.209417       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-a-02-18-21-1613652198-10-0-132-131-2-1 --id 10.0.130.185 --host 10.0.130.185 --client 10.132.0.0/14 --ikeport 4500 --to --id 10.0.132.131 --host 3.23.239.128 --client 172.30.0.0/16 --ikeport 4500]
I0218 17:02:37.213409       1 libreswan.go:337] Executing whack with args: [--psk --encrypt --forceencaps --name submariner-cable-majopela-cluster-a-02-18-21-1613652198-10-0-132-131-2-2 --id 10.0.130.185 --host 10.0.130.185 --client 10.132.0.0/14 --ikeport 4500 --to --id 10.0.132.131 --host 3.23.239.128 --client 10.128.0.0/14 --ikeport 4500]

We are checking with the libreswan team if:

1. Is there any other way in which we could configure the connection so that the packets could be identified properly regardless of the
   source IP and port? may be by using the configuration of pluto instead of whack (+Stephen Kitt was suggesting something like that)

side_a & side_b logs which from the submariner gateway containing the pluto debug output.:
side_a.gz
side_b.gz

[stale]

This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions.

[tpantelis]

bump

[jawabuu]

@raffaelespazzoli Interesting project.
Can keepalived-operator be used with cloud providers in cases where I do not want to the cloud configured load-balancer?

[raffaelespazzoli]

No because there is no way to disable the cloud provider behavior. Also many cloud providers don't let you create control the network and create ips.

…

On Sun, May 16, 2021, 12:46 AM jawabuu ***@***.***> wrote: @raffaelespazzoli <https://github.com/raffaelespazzoli> Interesting project. Can keepalived-operator be used with cloud providers in cases where I do not want to the cloud configured load-balancer? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1071 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPERXAPCM4HW3DMSFTKIGLTN5EYVANCNFSM4VVHZDBQ> .

[jawabuu]

@raffaelespazzoli

1. Most cloud-providers have a floating ip resource that can be moved across nodes via API.
2. A use case would be installing kubernetes using k3s or any other distros on cloud instances.

[raffaelespazzoli]

@jawabuu I don't understand where you are going with this reasoning. On major cloud providers (AWS, Azure, Google) the LoadBalancer Service API feature is provided directly by Kubernetes + cloud provider. The issue might exist on those Kubernetes deployment where there is no cloud provider or the provider does not implement the LoadBalancer Service API. In those instances the mentioned operators: MetalLB and keepalived can help. Either way in this context we assume that something is implementing the LoadBalancer service API and we are reasoning on how to leverage this fact for the submariner ingress traffic.

[jawabuu]

@raffaelespazzoli I do understand this. I'm aware that this is off track to the issue but it's my first encounter with keepalived-operator and was just exploring possibilities. : )
I'm actually focusing on cloud providers that do not configure cloud-loadbalancers/lack managed kubernetes services.
I will probably raise the query in that repo.
Could I reach you on slack?

[raffaelespazzoli]

you can contact me, sure.

…

On Sun, May 16, 2021 at 10:08 AM jawabuu ***@***.***> wrote: @raffaelespazzoli <https://github.com/raffaelespazzoli> I do understand this. I'm aware that this is off track to the issue but it's my first encounter with keepalived-operator and was just exploring possibilities. : ) I'm actually focusing on cloud providers that do not configure cloud-loadbalancers/lack managed kubernetes services. I will probably raise the query in that repo. Could I reach you on slack? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1071 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPERXDU4O5IWBWHVFCXI7LTN7GWNANCNFSM4VVHZDBQ> .

-- ciao/bye Raffaele

[mangelajo]

From the point of view of submariner, anything that will obey the LoadBalancer API, create a public IP and redirect UDP traffic to the submariner-gateway port it may work.

[jawabuu]

@mangelajo This is sufficient.

[github-actions]

🎉 Great news! Looks like all the dependencies have been resolved:

• Verify if ipsec-preferred server will work with a network LoadBalancer on AWS #1310

💡 To add or remove a dependency please update this issue/PR description.

Brought to you by Dependent Issues (:robot: ). Happy coding!

[stale]

This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions.