Restrict traffic to/from EKS Cluster

docs/eks-networking.md

@@ -0,0 +1,81 @@
+# Understanding VPC configuration for EKS in conjunction with Fargate (WIP*)
+
+*\*All of this is subject to change while this message is here.*
+
+Through work on [a security compliance issue](https://github.com/GSA/datagov-deploy/issues/3355), a thorough inspection of the networking design of this repo was completed.  By default, EKS clusters are fully publicly available.  The desire was to allow tighter configruation to prevent hacks/data leaks.  A combination of public+private networking is considered the [best practice for common Kubernetes workloads on AWS](https://aws.amazon.com/blogs/containers/de-mystifying-cluster-networking-for-amazon-eks-worker-nodes/) as it provides the flexibility of public availability alongside the security of private resources.
+
+Note: This repo utilizes Terraform to configure multiple intertwining parts from the AWS world to the Kubernetes world and wraps it up nicely with a Brokerpak bow.  Most of the concepts and commands are discussed in terms of Terraform, but there are AWS/K8S cli equivalents.
+
+Here's a non-detailed exhaustive list of modules/resources used:
+- Module [vpc](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/14.0.0)
+- Module [eks](https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/3.7.0)
+- Module [aws_load_balancer_controller](https://github.com/GSA/terraform-kubernetes-aws-load-balancer-controller)
+- Resource [aws_vpc_endpoint](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint)
+- Resource [aws_route53_resolver_endpoint](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route53_resolver_endpoint)
+
+## Desired Configuration/Design
+
+The user would like to provision a functional K8S cluster with the ability to host publicly-available application deployments.  The cluster will live in AWS Fargate to reduce the compliance burden of managing the security of node machines.
+
+### Deployment Stack:
+- Computing Levels of Abstraction:
+  - Fargate Nodes > EKS > Application
+- Networking Levels of Abstraction (the order is still being learned):
+  - Internal CIDRs (Private + Public) > Network ACLs > Security Groups > Ingress Controller > NAT Gateway > Application Load Balancer > Elastic IP (EIP) > Domain
+  - VPC > NAT Gateway > Load Balancer > Elastic IP (EIP)
+
+When a user accesses an application through the [domain](https://github.com/GSA/eks-brokerpak/blob/restrict-eks-traffic/terraform/provision/ingress.tf#L248-L254), it gets resolved to an EIP that gets routed to the [application load balancer](https://github.com/GSA/eks-brokerpak/blob/restrict-eks-traffic/terraform/provision/ingress.tf#L19-L31).  This then passes to the internal [ingress controller](https://github.com/GSA/eks-brokerpak/blob/restrict-eks-traffic/terraform/provision/ingress.tf#L36-L95) to the cluster nodes based on the [vpc configuration](https://github.com/GSA/eks-brokerpak/blob/restrict-eks-traffic/terraform/provision/vpc.tf#L30-L186).  That last step is intricate because the ingress controller lives within the VPC, so it only works if the VPC configuration permits.
+
+### Networking Design
+
+- The entire EKS cluster lives within the VPC (10.20.0.0/16).
+- There is a public subnet (10.20.101.0/24).
+- There is a private subnet (10.20.1.0/24).
+- The EKS control plane has a public endpoint (x.x.x.x/x).
+- The EKS control plane has a private endpoint (10.20.x.x/x).
+- Worker nodes, by default, communicate entirely on the private subnet.
+- The ingress controllers connect external traffic to worker nodes through the public subnet.
+- Security Groups and Network ACLs are used to control traffic.
+
+### Setting up Clusters in a Private Subnet
+
+In order for EKS to isolate workers in a private subnet, the following [VPC considerations](https://docs.aws.amazon.com/eks/latest/userguide/private-clusters.html) are necessary,
+
+- The VPC needs to configure private subnets,
+  - Define `private_subnets` cidrs.
+  - Set `enable_nat_gateway` (True) to allow ingresses to connect public and private subnets.
+  - Set `map_public_ip_on_launch` (False) to disable public ips being set on private subnets.
+  - Set `enable_dns_hostnames` and `enable_dns_support` to support DNS hostnames in the VPC (necessary for the [API Server](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html)).
+
+- The EKS needs to know about the VPC config,
+  - [Example](https://github.com/GSA/eks-brokerpak/blob/restrict-eks-traffic/terraform/provision/eks.tf#L15-L28)
+
+- VPC Endpoints are necessary for private cluster nodes to talk to other AWS services,
+  - [These are the ones](https://github.com/GSA/eks-brokerpak/blob/restrict-eks-traffic/terraform/provision/vpc.tf#L196-L265) I've identified as necessary for us,
+    - com.amazonaws.<region>.ec2
+    - com.amazonaws.<region>.ecr.api
+    - com.amazonaws.<region>.ecr.dkr
+    - com.amazonaws.<region>.s3 _– For pulling container images_
+    - com.amazonaws.<region>.logs _– For CloudWatch Logs_
+    - com.amazonaws.<region>.sts _– If using Cluster Autoscaler or IAM roles for service accounts_
+    - com.amazonaws.<region>.elasticloadbalancing _– If using Application Load Balancers_
+  - These are additional ones that may be necessary in the future,
+    - com.amazonaws.<region>.autoscaling _– If using Cluster Autoscaler_
+    - com.amazonaws.<region>.appmesh-envoy-management _– If using App Mesh_
+    - com.amazonaws.<region>.xray _– If using AWS X-Ray_
+
+- [Security Group (SG) Considerations](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html#cluster-sg)
+  - Control Plane
+    - Minimum Inbound traffic - 443/TCP from all node SGs
+    - Minimum Outbound traffic - 10250/TCP to all node SGs
+  - Nodes
+    - Minimum Inbound traffic - 10250/TCP from control plane SGs
+    - Minimum Outbound traffic - 443/TCP to control plane SGs
+
+- The [IAM Role](https://github.com/aws/amazon-vpc-cni-k8s/issues/30) needs to allow Nodes to pull images.
+  - Docs: https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-policy-examples.html
+  - [Terraform implementation](https://github.com/GSA/eks-brokerpak/blob/restrict-eks-traffic/terraform/provision/eks.tf#L150-L166) of creating the role policy
+
+- A [aws_route53_resolver_endpoint](https://github.com/GSA/eks-brokerpak/blob/restrict-eks-traffic/terraform/provision/vpc.tf#L14-L28) needs to be made available to the private subnet.
+
+- Careful consideration needs to be put towards the [user/roles](https://stackoverflow.com/questions/66996306/aws-eks-fargate-coredns-imagepullbackoff) for Fargate cluster creations.

eks-service-definition.yml

@@ -30,6 +30,18 @@ provision:
       pattern: ^[a-z0-9][a-z0-9-]*[a-z0-9]$
     default: ${str.truncate(64, "${request.instance_id}")}
     details: A subdomain to use for the cluster instance. Default is the instance ID.
+  - field_name: egress_allowed
+    required: false
+    type: array
+    details: "A list of IP ranges to allow egress traffic to (ex. [\"x.x.x.x/x\", ...])"
+    overwrite: true
+    default: null
+  - field_name: ingress_allowed
+    required: false
+    type: array
+    details: "A list of IP ranges to allow ingress traffic from (ex. [\"x.x.x.x/x\", ...])"
+    overwrite: true
+    default: null
   computed_inputs:
   - name: instance_name
     required: true
@@ -51,6 +63,7 @@ provision:
     default: ${config("aws.default_region")}
   - name: write_kubeconfig
     type: boolean
+    required: false
     overwrite: true
     default: false
   outputs:

network_policy/2048_fixture.yml

@@ -0,0 +1,53 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: deployment-2048
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: app-2048
+  replicas: 2
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: app-2048
+    spec:
+      containers:
+      - image: alexwhen/docker-2048
+        imagePullPolicy: Always
+        name: app-2048
+        ports:
+        - containerPort: 80
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: service-2048
+spec:
+  ports:
+    - port: 80
+      targetPort: 80
+      protocol: TCP
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: app-2048
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: ingress-2048
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.io/rewrite-target: /
+spec:
+  rules:
+  - http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: service-2048
+            port: 
+              number: 80              

network_policy/README.md

@@ -0,0 +1,39 @@
+Context:
+
+This sub-directory provides a test case for creating and applying network
+policies.  It..
+- Sets up a basic kind cluster (specifying the pod-network-cidr)
+- Installs an nginx ingress to make services available outside of the cluster
+- Installs calico as the network plugin to manage network policies
+
+The test case uses the 2048 service as a baseline of seeing network
+restrictions.  Different network policies are then applied to allow/restrict
+network traffic.
+
+Instuctions:
+
+To setup environment,
+
+`./startup.sh`
+
+To tear down envrionment,
+
+`./shutdown.sh`
+
+To create 2048 game,
+
+`kubectl apply -f 2048_fixture.yml`
+
+To apply network policy,
+
+`kubectl apply -f test_deny.yml`
+
+To test egress traffic,
+
+`kubectl exec -it pod/<2048-pod> -- sh -c "ping -c 4 8.8.8.8"`
+
+To test ingress traffic,
+
+Visit the 2048 game, default url is http://default-http-backend/
+Note: Make sure to add the host to ip translation in /etc/hosts or similar
+`127.0.0.1 default-http-backend`

network_policy/kind-config.yaml

@@ -0,0 +1,29 @@
+# four node (three workers) cluster config
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+networking:
+  podSubnet: "10.0.0.0/8"
+nodes:
+- role: control-plane
+  image: kindest/node:v1.19.11@sha256:07db187ae84b4b7de440a73886f008cf903fcf5764ba8106a9fd5243d6f32729
+  # Mapping an ingress controller to host ports
+  # See docs at https://kind.sigs.k8s.io/docs/user/ingress/#create-cluster
+  kubeadmConfigPatches:
+  - |
+    kind: InitConfiguration
+    nodeRegistration:
+      kubeletExtraArgs:
+        node-labels: "ingress-ready=true"
+  extraPortMappings:
+  - containerPort: 80
+    hostPort: 80
+    protocol: TCP
+  - containerPort: 443
+    hostPort: 443
+    protocol: TCP
+- role: worker
+  image: kindest/node:v1.19.11@sha256:07db187ae84b4b7de440a73886f008cf903fcf5764ba8106a9fd5243d6f32729
+- role: worker
+  image: kindest/node:v1.19.11@sha256:07db187ae84b4b7de440a73886f008cf903fcf5764ba8106a9fd5243d6f32729
+- role: worker
+  image: kindest/node:v1.19.11@sha256:07db187ae84b4b7de440a73886f008cf903fcf5764ba8106a9fd5243d6f32729

network_policy/shutdown.sh

@@ -0,0 +1 @@
+kind delete cluster --name datagov-broker-test

network_policy/startup.sh

@@ -0,0 +1,12 @@
+# Creating a temporary Kubernetes cluster to test against with KinD
+kind create cluster --config kind-config.yaml --name datagov-broker-test
+
+# Install a KinD-flavored ingress controller (to make the Solr instances visible to the host).
+# See (https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx for details.
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.1/deploy/static/provider/kind/deploy.yaml	
+kubectl wait --namespace ingress-nginx \
+    --for=condition=ready pod \
+    --selector=app.kubernetes.io/component=controller \
+    --timeout=270s
+
+kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

network_policy/test_deny.yml

@@ -0,0 +1,9 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: ingress-default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+  - Ingress
+  - Egress

terraform/provision/Dockerfile

@@ -5,6 +5,6 @@ FROM alpine/k8s:1.20.7
 COPY --from=terraform /bin/terraform /bin/terraform 
 
 RUN apk update
-RUN apk add --update git 
+RUN apk add --update git bind-tools
 
 ENTRYPOINT ["/bin/sh"]

terraform/provision/crds.tf

@@ -7,11 +7,11 @@
 # solr-operator do it so that it will register and unregister its CRDs as part
 # of the helm install process.
 resource "helm_release" "zookeeper-operator" {
-  name             = "zookeeper"
-  chart            = "zookeeper-operator"
-  repository       = "https://charts.pravega.io/"
-  version          = "0.2.12"
-  namespace        = "kube-system"
+  name       = "zookeeper"
+  chart      = "zookeeper-operator"
+  repository = "https://charts.pravega.io/"
+  version    = "0.2.12"
+  namespace  = "kube-system"
   set {
     # See https://github.com/pravega/zookeeper-operator/issues/324#issuecomment-829267141
     name  = "hooks.delete"