EdgeCase: docker-mailserver on bare-metal kubernetes with metallb as loadbalancer

[tibeer]

Hi there!

I know my scenario is kind of an edge-case, but maybe someone can help me. I am running Kubernetes on bare metal as a single node cluster. The machine itself has one public IP natted to its private IP address (e.g. 1.2.3.4 is natted to 10.0.0.20). Thus, my metallb ipaddresspool looks like this:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  creationTimestamp: "2024-06-22T05:05:20Z"
  generation: 1
  name: pool
  namespace: metallb-system
  resourceVersion: "1033"
  uid: b545786c-a08b-4e8e-a643-72c35a0c837c
spec:
  addresses:
  - 10.0.0.20/32
  autoAssign: true
  avoidBuggyIPs: false

This works fine for all applications that use a service of type LoadBalancer with externalTrafficPolicy: Cluster, but this is of course not working for docker-mailserver, as it requires the Local externalTrafficPolicy. Since I only have one public IP available in my metallb Pool, I need to use certain metallb annotations (allow-shared-ip) to share the IP between services:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: docker-mailserver
    meta.helm.sh/release-namespace: docker-mailserver
    metallb.universe.tf/address-pool: pool
    metallb.universe.tf/allow-shared-ip: default
    prometheus.io/path: /metrics
    prometheus.io/port: "9102"
    prometheus.io/probe: "false"
    prometheus.io/scrape: "true"
  creationTimestamp: "2024-09-10T05:06:39Z"
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: docker-mailserver
    chart: docker-mailserver-4.0.5
    heritage: Helm
    release: docker-mailserver
  name: docker-mailserver
  namespace: docker-mailserver
  resourceVersion: "22750686"
  uid: 040776d5-ac86-4a96-ad07-50425d46eaaf
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.96.49.23
  clusterIPs:
  - 10.96.49.23
  externalTrafficPolicy: Local
  healthCheckNodePort: 32252
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: smtp
    nodePort: 31950
    port: 25
    protocol: TCP
    targetPort: smtp
  - name: submissions
    nodePort: 31416
    port: 465
    protocol: TCP
    targetPort: submissions
  - name: submission
    nodePort: 31180
    port: 587
    protocol: TCP
    targetPort: submission
  - name: smtp-proxy
    nodePort: 31784
    port: 12525
    protocol: TCP
    targetPort: smtp-proxy
  - name: subs-proxy
    nodePort: 32593
    port: 10465
    protocol: TCP
    targetPort: subs-proxy
  - name: sub-proxy
    nodePort: 31094
    port: 10587
    protocol: TCP
    targetPort: sub-proxy
  - name: imap
    nodePort: 31984
    port: 143
    protocol: TCP
    targetPort: imap
  - name: imaps
    nodePort: 31504
    port: 993
    protocol: TCP
    targetPort: imaps
  - name: imap-proxy
    nodePort: 30708
    port: 10143
    protocol: TCP
    targetPort: imap-proxy
  - name: imaps-proxy
    nodePort: 30947
    port: 10993
    protocol: TCP
    targetPort: imaps-proxy
  - name: rspamd
    nodePort: 32494
    port: 11334
    protocol: TCP
    targetPort: rspamd
  selector:
    app.kubernetes.io/name: docker-mailserver
    release: docker-mailserver
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

But since the externalTrafficPolicy is required to be local for docker-mailserver, this conflicts with sharing the IP. Has anybody an idea how I can solve that situation?

[xorinzor]

You should increase the amount of ip-adresses in your pool. Since it's LAN, there's no reason to limit yourself to 1 IP.
Then you can still choose which ports should be NATted.

[tibeer]

How would I do that? My understanding is, that 10.0.0.20 is the only local IP address that gets natted out. Unfortunately I cannot set wich IP is natted, as it is handled by the cloud provider for me. So increasing the pool would result in most services not being reachable from the internet, except of those that are bound to 10.0.0.20. Or am I missing something here?

[xorinzor]

You didn't include that information in your original post.

Unfortunately there's no way to bind to a port twice on the same IP.

[tibeer]

Whops, sorry for that. Fore sure you cannot bind the same port twice to the same IP. But there is no other service running that uses these ports. I can deploy the helm chart without any problem, but the service itself cannot connect to the outside world because of the externalTrafficPolicy.

[xorinzor]

Ah I misunderstood the issue.

From what I can tell this is a hard limitation: metallb/metallb#271 (comment)

[tibeer]

Ah, too bad. But thanks a lot for linking the issue in metallb :)