Across clusters without discovery when a pod connects via TCP to a remote pod when the pod is not on a gateway and the remote pod is on a gateway

[NickPak]

What happened:
I tried to install Submariner in two Kubernetes clusters when I realized that when communicating across the clusters over TCP, the Pod in the non-gateway node cannot communicate with the Pod in the other cluster. Then, I tried to test the Kind-based cluster as per the documentation and also found the same issue as mentioned above, I tried many versions: 0.19.0-m3, 0.18.0, 0.14.9, and all of them have this issue. However, two Pods that are at the gateway node at the same time can communicate normally.

What you expected to happen:
I want that Pods across clusters can communicate with each other arbitrarily via TCP, they don't go through DNS service discovery and connect directly via IP address. It doesn't matter if this Pod is on a gateway node, or a non-gateway node, it can communicate normally.
How to reproduce it (as minimally and precisely as possible):
Follow this document to reproduce: https://submariner.io/getting-started/quickstart/kind/
Anything else we need to know?:

Environment:

• Diagnose information (use subctl diagnose all):
  Cluster "cluster2"
  ✓ Checking Submariner support for the Kubernetes version
  ✓ Kubernetes version "v1.31.0" is supported

✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
✓ Checking DaemonSet "submariner-gateway"
✓ Checking DaemonSet "submariner-routeagent"
✓ Checking DaemonSet "submariner-metrics-proxy"
✓ Checking Deployment "submariner-lighthouse-agent"
✓ Checking Deployment "submariner-lighthouse-coredns"
✓ Checking the status of all Submariner pods
✓ Checking that gateway metrics are accessible from non-gateway nodes

✓ Checking Submariner support for the CNI network plugin
✓ The detected CNI network plugin ("kindnet") is supported
✓ Checking gateway connections
✓ Checking Submariner support for the kube-proxy mode
✓ The kube-proxy mode is supported
✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

✓ Checking that services have been exported properly

Cluster "cluster1"
✓ Checking Submariner support for the Kubernetes version
✓ Kubernetes version "v1.31.0" is supported

✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
✓ Checking DaemonSet "submariner-gateway"
✓ Checking DaemonSet "submariner-routeagent"
✓ Checking DaemonSet "submariner-metrics-proxy"
✓ Checking Deployment "submariner-lighthouse-agent"
✓ Checking Deployment "submariner-lighthouse-coredns"
✓ Checking the status of all Submariner pods
✓ Checking that gateway metrics are accessible from non-gateway nodes

✓ Checking Submariner support for the CNI network plugin
✓ The detected CNI network plugin ("kindnet") is supported
✓ Checking gateway connections
✓ Checking Submariner support for the kube-proxy mode
✓ The kube-proxy mode is supported
✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

✓ Checking that services have been exported properly

• Gather information (use subctl gather):
  Cluster "cluster1"
  Gathering information from cluster "cluster1"
  ✓ Gathering connectivity logs
  ✓ Found 1 pods matching label selector "app=submariner-gateway"
  ✓ Found 2 pods matching label selector "app=submariner-routeagent"
  ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"
  ✓ Found 0 pods matching label selector "app=submariner-globalnet"
  ✓ Found 0 pods matching label selector "app=submariner-addon"
  ✓ Gathering connectivity resources
  ✓ Gathering CNI data from 2 pods matching label selector "app=submariner-routeagent"
  ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"
  ✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"
  ✓ Found 2 endpoints in namespace "submariner-operator"
  ✓ Found 2 clusters in namespace "submariner-operator"
  ✓ Found 1 gateways in namespace "submariner-operator"
  ✓ Found 0 clusterglobalegressips in namespace ""
  ✓ Found 0 globalegressips in namespace ""
  ✓ Found 0 globalingressips in namespace ""
  ✓ Gathering service-discovery logs
  ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
  ✓ Found 2 pods matching label selector "k8s-app=kube-dns"
  ✓ Gathering service-discovery resources
  ✓ Found 0 serviceexports in namespace ""
  ✓ Found 0 serviceimports in namespace ""
  ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
  ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
  ✓ Found 1 configmaps by field selector "metadata.name=coredns" in namespace "kube-system"
  ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
  ✓ Gathering broker logs
  ✓ Gathering broker resources
  ✓ Found 2 endpoints in namespace "submariner-k8s-broker"
  ✓ Found 2 clusters in namespace "submariner-k8s-broker"
  ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"
  ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"
  ✓ Gathering operator logs
  ✓ Found 1 pods matching label selector "name=submariner-operator"
  ✓ Gathering operator resources
  ✓ Found 1 submariners in namespace "submariner-operator"
  ✓ Found 1 servicediscoveries in namespace "submariner-operator"
  ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
  ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
  ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
  ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
  ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
  ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
  ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
  ✓ Found 0 services by field selector "metadata.name=submariner-gateway" in namespace "submariner-operator"
  Files are stored under directory "submariner-20240907083645/cluster1"

Cluster "cluster2"
Gathering information from cluster "cluster2"
✓ Gathering connectivity logs
✓ Found 1 pods matching label selector "app=submariner-gateway"
✓ Found 2 pods matching label selector "app=submariner-routeagent"
✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"
✓ Found 0 pods matching label selector "app=submariner-globalnet"
✓ Found 0 pods matching label selector "app=submariner-addon"
✓ Gathering connectivity resources
✓ Gathering CNI data from 2 pods matching label selector "app=submariner-routeagent"
✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"
✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"
✓ Found 2 endpoints in namespace "submariner-operator"
✓ Found 2 clusters in namespace "submariner-operator"
✓ Found 1 gateways in namespace "submariner-operator"
✓ Found 0 clusterglobalegressips in namespace ""
✓ Found 0 globalegressips in namespace ""
✓ Found 0 globalingressips in namespace ""
✓ Gathering service-discovery logs
✓ Found 3 pods matching label selector "component=submariner-lighthouse"
✓ Found 2 pods matching label selector "k8s-app=kube-dns"
✓ Gathering service-discovery resources
✓ Found 0 serviceexports in namespace ""
✓ Found 0 serviceimports in namespace ""
✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
✓ Found 1 configmaps by field selector "metadata.name=coredns" in namespace "kube-system"
✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
✓ Gathering broker logs
✓ Gathering broker resources
✓ Found 2 endpoints in namespace "submariner-k8s-broker"
✓ Found 2 clusters in namespace "submariner-k8s-broker"
✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"
✓ Found 0 serviceimports in namespace "submariner-k8s-broker"
✓ Gathering operator logs
✓ Found 1 pods matching label selector "name=submariner-operator"
✓ Gathering operator resources
✓ Found 1 submariners in namespace "submariner-operator"
✓ Found 1 servicediscoveries in namespace "submariner-operator"
✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
✓ Found 0 services by field selector "metadata.name=submariner-gateway" in namespace "submariner-operator"
Files are stored under directory "submariner-20240907083645/cluster2"

• Cloud provider or hardware configuration:
  Debian 11.4 amd64
  Docker
  Engine:
  Version: 27.2.0
  API version: 1.47 (minimum version 1.24)
  Go version: go1.21.13
  Git commit: 3ab5c7d
  Built: Tue Aug 27 14:15:27 2024
  OS/Arch: linux/amd64
  Experimental: false
  containerd:
  Version: 1.7.21
  GitCommit: 472731909fa34bd7bc9c087e4c27943f9835f111
  runc:
  Version: 1.1.13
  GitCommit: v1.1.13-0-g58aa920
  docker-init:
  Version: 0.19.0

kind v0.24.0 go1.22.6 linux/amd64

• Install tools:
  subctl 0.19.0-m3/0.18.0/0.14.9
• Others:
  Sep 7 16:07:45.788: Validating that dig result are not ""
  Sep 7 16:07:45.788: Deleting serviceExport nginx-ss.e2e-tests-discovery-dfcfw on "cluster2"
  Sep 7 16:07:45.792: Retrieving ServiceImport for "nginx-ss" in ns "e2e-tests-discovery-dfcfw" on "cluster1"
  Sep 7 16:07:46.294: Deleting namespace "e2e-tests-discovery-dfcfw" on cluster "cluster1"
  Sep 7 16:07:46.297: Deleting namespace "e2e-tests-discovery-dfcfw" on cluster "cluster2"
  Sep 7 16:07:46.300: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-dfcfw" on "cluster2"
  Sep 7 16:07:46.302: Retrieving EndpointSlices for "" in ns "e2e-tests-discovery-dfcfw" on "cluster1"
  • [40.522 seconds]

---

Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod when the pod is not on a gateway and the re mote pod is not on a gateway should have sent the expected data from the pod to the other pod [dataplane, basic]
github.com/submariner-io/[email protected]/test/e2e/dataplane/tcp_pod_connectivity.go:38
Sep 7 16:07:46.304: Creating namespace objects with basename "dataplane-conn-nd"
Sep 7 16:07:46.308: Generated namespace "e2e-tests-dataplane-conn-nd-qnsxj" in cluster "cluster1" to execute the tests in
Sep 7 16:07:46.308: Creating namespace "e2e-tests-dataplane-conn-nd-qnsxj" in cluster "cluster2"
Sep 7 16:07:46.313: Creating a listener pod in cluster "cluster2", which will wait for a handshake over TCP
Sep 7 16:07:47.822: Sep 7 16:07:47.822: INFO: Will send traffic to IP: 10.131.1.11

Sep 7 16:07:47.822: Creating a connector pod in cluster "cluster1", which will attempt the specific UUID handshake over TCP
Sep 7 16:07:47.829: Waiting for the connector pod "tcp-check-podnf7dt" to exit, returning what connector sent
Sep 7 16:10:49.832: Sep 7 16:10:49.832: INFO: Pod "tcp-check-podnf7dt" on node "cluster1-worker" output:
nc: 10.131.1.11 (10.131.1.11:1234): Operation timed out
nc: 10.131.1.11 (10.131.1.11:1234): Operation timed out

Sep 7 16:10:49.832: Waiting for the listener pod "tcp-check-listenerr6ppk" to exit, returning what listener sent
Sep 7 16:10:49.838: Sep 7 16:10:49.838: INFO: Pod "tcp-check-listenerr6ppk" on node "cluster2-worker" output:
listening on 0.0.0.0:1234 ...
nc: timeout

Sep 7 16:10:49.838: Sep 7 16:10:49.838: INFO: Connector pod has IP: 10.130.1.10

[FAILED] in [It] - github.com/submariner-io/[email protected]/test/e2e/framework/network_pods.go:196 @ 09/07/24 16:10:49.838
Sep 7 16:10:49.838: Deleting namespace "e2e-tests-dataplane-conn-nd-qnsxj" on cluster "cluster1"
Sep 7 16:10:49.841: Deleting namespace "e2e-tests-dataplane-conn-nd-qnsxj" on cluster "cluster2"
• [FAILED] [183.540 seconds]
Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod when the pod is not on a gateway and the re mote pod is not on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, basic]
github.com/submariner-io/[email protected]/test/e2e/dataplane/tcp_pod_connectivity.go:38

[FAILED] Expected
: 1
to equal
: 0
In [It] at: github.com/submariner-io/[email protected]/test/e2e/framework/network_pods.go:196 @ 09/07/24 16:10:49.838

Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod when the pod is not on a gateway and the re mote pod is on a gateway should have sent the expected data from the pod to the other pod [dataplane]
github.com/submariner-io/[email protected]/test/e2e/dataplane/tcp_pod_connectivity.go:38
Sep 7 16:10:49.844: Creating namespace objects with basename "dataplane-conn-nd"
Sep 7 16:10:49.848: Generated namespace "e2e-tests-dataplane-conn-nd-qtclf" in cluster "cluster1" to execute the tests in
Sep 7 16:10:49.848: Creating namespace "e2e-tests-dataplane-conn-nd-qtclf" in cluster "cluster2"
Sep 7 16:10:49.852: Creating a listener pod in cluster "cluster2", which will wait for a handshake over TCP
Sep 7 16:10:50.861: Sep 7 16:10:50.861: INFO: Will send traffic to IP: 10.131.0.6

Sep 7 16:10:50.861: Creating a connector pod in cluster "cluster1", which will attempt the specific UUID handshake over TCP
Sep 7 16:10:50.868: Waiting for the connector pod "tcp-check-pod9w7xn" to exit, returning what connector sent

[yboaron]

Thanks for reaching out @NickPak ,

1. For Kind deployment, we have encountered cases in the past where Kind with default CNI (KindNet) did not work for some environments.

can you try with cni=OVN ? you can use

make deploy using=lighthouse,ovn from submariner-operator repo.

2. Can you elaborate on your non-Kind K8S clusters? platform? cni ?

[gk-fschubert]

We've the same issue and I wanted to open a ticket as well.

Our current setup is:

• 2 k8s clusters, hosted on DigitalOcean(cilium as CNI), k8s version 1.28.2
• globalnet used
• submariner uses the generic CNI plugin

[yboaron]

We've the same issue and I wanted to open a ticket as well.

Our current setup is:

• 2 k8s clusters, hosted on DigitalOcean(cilium as CNI), k8s version 1.28.2
• globalnet used
• submariner uses the generic CNI plugin

Hi @gk-fschubert ,
Please file a separate issue for your case.
Attach subctl gather [1] directories for both clusters

[1]
subctl gather --kubeconfig <cluster_kubeconfig>

[NickPak]

Version 0.15.5 works fine in my test environment.

[NickPak]

In addition, in a managed cloud Kubernetes environment, if you need to directly access a Pod in another cluster via an IP address, you need to open up the underlying network between multiple clusters.

[yboaron]

In addition, in a managed cloud Kubernetes environment, if you need to directly access a Pod in another cluster via an IP address, you need to open up the underlying network between multiple clusters.

Were you able to allow inter-cluster connectivity using Submariner ?
Can you elaborate on your managed Kubernetes environment (including CNI) ?

Also, can you elaborate on you need to open up the underlying network between multiple clusters ? do you mean to allow inter-cluster traffic in security-group (SRC IP= < some IP from cluster1 pod CIDR >, dest IP= < IP from cluster2 pod CIDR >) ?