The network policy behavior in k3s is not performing as expected.

[Mr-lq7]

Environmental Info:
K3s Version: v1.28.8-k3s1

Node(s) CPU architecture, OS, and Version:

host machine: Linux #1 SMP Sat Oct 7 17:52:50 CST 2023 x86_64 x86_64 x86_64 GNU/Linux
k3s container machine: Linux #1 SMP Sat Oct 7 17:52:50 CST 2023 x86_64 GNU/Linux

Cluster Configuration:

I launched k3s using docker-compose.yaml .

Describe the bug:

When I apply the network policy, the busybox pod in the default namespace can access the svc in default namespace except the API Server SVC(kubernetes svc).

Steps To Reproduce:

• Installed K3s:
  the docker-compose.yaml is as follows：

version: "3.4"

services:

  k3s:
    hostname: k3s
    image: rancher/k3s:v1.28.8-k3s1
    tty: true
    stdin_open: true
    entrypoint: 
      - /bin/sh
      - -c
      - >-
        ip=$$(nslookup k3s | awk ' /^Address: / { print $2 } ');
        k3s server --bind-address $$ip --advertise-address $$ip
    command: "" 
    tmpfs:
      - /run
      - /var/run

    ulimits:
      nproc: 65536
      nofile:
        soft: 65536
        hard: 65536
    privileged: true
    restart: always
    volumes:
      - ./tmp/k3s-master:/var/lib/k3s
      - ./tmp/k3s-master-node:/etc/rancher/node/
      - ./tmp/kube:/kube
    expose:
      - 6443
      - 80
      - 443
    environment:
      - K3S_TOKEN=${K3S_TOKEN:-k3s_token}
      - K3S_KUBECONFIG_OUTPUT=/kube/config
      - K3S_KUBECONFIG_MODE=600

1. run the command: docker-compose up -d
2. run the command: docker-compose exec k3s sh
3. my network policy(netpol.yaml) is as follows:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-basic-service
  namespace: sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 #safe agent的工作namespace
spec:
  podSelector: {}
  egress:
    - to:
      - namespaceSelector:
          matchExpressions:
            - key: kubernetes.io/metadata.name
              operator: In
              values:
                - default
  policyTypes:
    - Egress

4. my ns.yaml is as follows:

apiVersion: v1
kind: Namespace
metadata:
  name: sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9

5. run the command: kubectl apply -f ns.yaml
6. run the command: kubectl get ns

7. run the command: kubectl -n sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 run busybox --image=busybox -- /bin/sh -c "sleep 36000"

8. run the command: kubectl get svc

9. run the command: kubectl -n sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 exec -it busybox -- telnet 10.43.0.1 443

10. run the command: kubectl -n sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 exec -it busybox -- telnet 10.43.128.218 9000

11. run the command: kubectl apply -f netpol.yaml
12. run the command: kubectl -n sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 get netpol

13. run the command: kubectl -n sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 exec -it busybox -- telnet 10.43.0.1 443

14. run the command: kubectl -n sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 exec -it busybox -- telnet 10.43.128.218 9000

Expected behavior:

From the results of the screenshots at step 9 and step 10, the busybox pod in sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 namespace can access the API Server SVC(10.43.0.1) and the minio svc(10.43.128.218)；

After applying the netpol.yaml, the busybox pod in sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 namespace should be able to access the all svc in default namespace(including API Server SVC);

Actual behavior:

But after applying the netpol.yaml, the busybox pod in sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 namespace can access the minio svc(step 14), but can not access the API Server SVC(10.43.0.1) (step 13). The minio svc and the API Server SVC are both in default namespace!

Additional context / logs:

there is no other netpolicy;

I really want to know why the busybox pod in sandbox-73e2db8d-3310-4822-a447-47c0dc2711e9 namespace cannot access the API Server SVC, and how can I modify the netpol.yaml so that the pod can access all services in default namespace? I’ve been puzzled for a long time， thanks a lot!

[brandond]

cc @manuelbuil

[manuelbuil]

Thanks for the detailed explanation. The kubernetes service is a bit special because it is pointing to the nodeIP and not to a typical pod IP. If you query kubectl get endpoints , you'll see the IP of "real" process implementing it, which is the same as the node.

To make it work, additionally to your current network policy, you should use an ipBlock including the IP of the node to allow that communication. This is the typical way to grant communication with hostNetwork pods in egress network policies

[Mr-lq7]

It works! Thanks a lot!