Nodeport and Network Policies

[prebhakta]

Describe the problem/challenge you have
I have external (known) clients to the private Kubernetes cluster that I run that need access to some of the pods in my cluster. To simplify the process, I want to expose these pods via nodeport and by using other tools, I can create a Service Registry on these exposed pods for my clients to consume. I would like to secure this solution using network policies so that I can have a default deny all policy but allow ingress traffic from my external clients only.

Describe the solution you'd like
Looking at Calico, it seems there is a feature that covers this use-case. I imagine that an Antrea solution could be similar but I'm open to other ideas.

Anything else you would like to add?
I appreciate the help.

[antoninbas]

This is similar to something that came up in the past: #280

@abhiraut is this a feature that's on your radar?

[abhiraut]

yes. but not scoped for the next few releases at least. @jianjuns mentioned it also needs NodePort support with AntreaProxy before we could implement such network policies with Antrea.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment, or this will be closed in 180 days

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment, or this will be closed in 180 days

[jianjuns]

@salv-orlando @GraysonWu @vicky-liu : there are customers requesting this feature. Could we evaluate and decide a release plan?

[GraysonWu]

If I understand this issue correctly, we want our network policy could control the external access to a NodePort Service. I have some draft ideas, share them here:

First of all, in most cases, users are using IPBlock to select theirs external clients. Then we should enforce our network policy before any SNAT happens in order to correctly do the source matching. Antrea proxyAll doesn't perform SNAT before sending NodePort traffic to OVS. The first SNAT could happen in SNAT table on the first Node of the whole traffic path when the endpoint of the Service is on the other Node. This means to correctly do source matching we should enforce network policy on the first Node before SNAT table.

Antrea proxyAll will perform DNAT before sending NodePort traffic to OVS, changing the destination IP to a virtual IP. So for the destination matching, we could leverage our toServices implementation using the Service group ID to do the matching. Because no matter the client is inside the cluster or outside the cluster and the client use Cluster IP or NodePort to access the Service, the packet will go into theServiceLB table and load the service group ID in to the register on the first Node. This means to correctly do destination matching we should enforce network policy on the first Node after ServiceLB table.

Unfortunitly, ingress tables are tables after SNAT table. There are only two eligible stages: EgressSecurity and Routing. I have some thoughts:

1. Load the original IP into register in SNAT table, so that we could use the register to do source matching in ingress tables
2. Convert ingress rule to egress rule in controller then we could enforce network policy in egress tables
3. Add a something like preIngressSecurity stage before Routing stage

For me, I think 1. will be better. 2 & 3 are kind mess around our implementation by this special case. Glad to hear your opinions and other good ideas. Thanks! @tnqn @jianjuns @hongliangl @salv-orlando

For the API-wise, I think policies like below looks good?
We add serviceReference(or other name) field in appliedTo. And when processing this network policy, like toServices, we also use the service group ID to match the destination. And if we choose idea 1 above, then if for applied to service rule we use register to match the source.

apiVersion: crd.antrea.io/v1alpha1
kind: ClusterNetworkPolicy
metadata:
  name: default-deny-all
spec:
  priority: 1
  tier: Baseline
  appliedTo:
    - serviceReference:
        name: svc-a
        ns: ns-a
  ingress: 
    - action: Drop

apiVersion: crd.antrea.io/v1alpha1
kind: ClusterNetworkPolicy
metadata:
  name: allow-my-client
spec:
  priority: 1
  tier: Application
  appliedTo:
    - serviceReference:
        name: svc-a
        ns: ns-a
  ingress:
    - action: Allow
      ipBlock:
        cidr: x.x.x.x/32        

[jianjuns]

@GraysonWu Your approach 1) sounds good to me. For API, probably we should explicitly call out it is NodePort Service (e.g. changing serviceReference to nodePortService).

[tnqn]

1 makes more sense to me, ct_nw_src already has the original IP, perhaps we could use it directly?

For API, I wonder if this approach can work with ClusterIP, LoadbalancerIPs too. I guess yes. Should we just name it service, even if we only support matching NodePort traffic for now?

[jianjuns]

Humm, it might make sense to Service of LoadBalancer type too when using with ExternalIP management or MetalLB, probably not for ClusterIP though. I am fine with service too.

[GraysonWu]

1 makes more sense to me, ct_nw_src already has the original IP, perhaps we could use it directly?

Oh, if this is the case, then it will be easier.

For API, I wonder if this approach can work with ClusterIP, LoadbalancerIPs too. I guess yes. Should we just name it service, even if we only support matching NodePort traffic for now?

Yes, since no matter what type of Service, with AntreaProxy, we can use its groupID to match traffic.