Configurable subnet/security group per pod?

[mumoshu]

Hey! Thanks as always for providing us the world's best cloud 😉

This is initially a question but can be turned into a feature request according to further discussions.

#165 has added the feature that works like "configurable pod subnet/security group per node". Would it be beneficial and feasible to extend it to be able to configure it per pod?

My use-case for such feature is minimizing EC2 instance cost by further putting pods workloads together. What we have today works by creating an ASG of nodes per ENIConfig(=desired set of pod subnet/security groups), which results in either reduced intensity of workloads per node or increased complexity of the cluster setup due to varying EC2 instance types.

My guess was that you would suggest varying EC2 instance types. That is, just create an ENIConfig per a dedicated pod subnet+security group set, while also using smaller instance types for less used pod subnet+security groups set. But I thought it would still worth clarifying by writing here as an issue anyway.

[asevans]

I also would greatly prefer the ability to scope security groups to pods. I need to restrict network access from the cluster to non-EKS resources except for workloads that are allowed (I don't want to have to restrict pods to specific nodes, or do anything that prevents me from flexibly modifying the relationship between workloads I'm running).

[mumoshu]

Your point about flexibly modifying the relationship between workloads, perhaps by not forcing us to manually provision ASG per ENIConfig as I understood, makes sense.
Thanks for chiming in!

[andrh]

I'm also very interested in having this as a feature if it's possible.

[ghost]

Configuration could be done by a set of annotations on the pod itself.

Similar how containerd/CRI does to switch between container runtime (kata/gvisor etc) : https://github.com/containerd/cri/blob/master/pkg/annotations/annotations.go

[garo]

A pod could have an annotation describing the required security group id. Based on this the CNI would create a new ENI with the appropriate security group and assign the pod an ip from this ENI.

As a single ENI can support multiple IPs (depending on instance type from 4 to 50) If another pod requires the same security group the pod could get an IP from the same ENI, but in other case the ENI would just be not filled completely and this would be an acceptable tradeoff.

I guess that this would need to be taken in account with the pod scheduling system (Pod Capacity) so that a pod wouldn't be scheduled into an node which could not give an ip due to the security group settings.

[ofiliz]

Thank you all for the feedback! We are always working on improving container networking in EC2. We'll have more to share in the future.

[s1rc]

Subscribing. Definitely looking forward to being able to assign security groups to pods.

[C0en]

Apart from being able to assign a seperate config to a pod I think the most useful option would be to be able to assign a eniconfig per namespace.

[pwdebruin1]

Yeah definitely a feature to have from a security perspective, currently to isolate pods from communicating with each other the only workaround I can think off is using custom ENIConfig with certain nodes and then using a combination of taints, nodeSelectors and nodeAffinity to ensure HA with the correct pods assigned to the correct nodes with a security groups only appropriate for them. However this will leave nodes possibly under-scheduled. Does anyone use any other work arounds at the moment that proved helpful?

[badping]

This feature would help me aswell.
I have many pods on one node and want to restrict pod access to rds database.

[mr-karan]

bump
+1 for this feature request. This would really be helpful for us.

[ahrkrak]

As mentioned on related issue: aws/containers-roadmap#177
It is possible to apply Security Groups at the pod level today with amazon-vpc-cni using Tigera Secure Cloud Edition:
On Marketplace here: https://aws.amazon.com/marketplace/pp/B07FT3NTWX
Free trial link: https://tigera.io/free-ce

[joan-s-molas]

Very nice but I think this should be a native feature in the CNI at some point and people shouldn't have to rely on third-party software which will only add unnecessary complexity and extra possible points of failure.

[JessieAMorris]

I'm very interested in this as well. It seems surprising to me that this isn't a feature already.

[mogren]

Right now, if you have a new EKS 1.17 eks.3 cluster and CNI v1.7.1 or later, setting ENABLE_POD_ENI=true should enable this. See the documentation for details:

https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html