Ensure all inter-pod traffic uses TLS

[mogul]

User Story

In order to have TLS on every network hop between the outside world and individual pods, we want EKS clusters configured to use AWS App Mesh and cert-manager.

Acceptance Criteria

• GIVEN I have provisioned an EKS instance
  AND I have deployed the 2048 fixture
  AND I have accessed the 2048 application using my browser
  WHEN I run kubectl -n default exec -it ${2048_POD_NAME} -c envoy -- curl -s localhost:9901/stats | grep ssl.handshake
  THEN I see a non-zero count of ssl_handshake entries between the 2048 pod and the nginx-ingress pod.

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

This work will help us meet our compliance requirements. See section 10.9.6.

Sketch

For this story, we only need to work up through step 4.1 of the referenced blog post... That is, we want to demonstrate mTLS between the nginx-ingress pod and the 2048 pod.

We can work up through step 5 (TLS between the ALB controller and nginx-ingress controller) in a separate/future story.

We're now considering 4 options going forward:

1. Remove nginx-ingress to get as close to the AWS-supported configuration as possible (adds ALB costs)
2. Try the new solr-operator support for inter-node TLS (solves for Solr, further work needed in future for other k8s services)
3. Try the AWS+Kong documented method that uses Kong as the ingress controller (keeps single ALB)
4. Keep trying to debug existing path

See also https://docs.aws.amazon.com/app-mesh/latest/userguide/getting-started-kubernetes.html

[mogul]

We have to figure out why our annotations aren't making it all the way from the helm_release resource to the pods.

[adborden]

As we're re-evaluating our approach, I wanted to mention another option, the Solr Operator has support for mutual TLS. It doesn't look simple to implement and it doesn't look like it covers ZooKeeper but it might end up being an overall simpler solution that meets the compliance requirements.

[mogul]

It appears this is new functionality as of version 0.3.0 of solr-operator, which is not yet released. However, it's due any day now! Here's the 6-day-old ChangeLog.

[mogul]

BTW, I think the inter-pod TLS functionality is relatively straightforward if cert-manager is installed in the k8s cluster. It's the mTLS (client-certificate) part that is more complex.

[mogul]

Srini is examining this alternate approach.

[mogul]

For the record: it turns out that if you're using Fargate, AWS AppMesh is the only viable service mesh option for now. Brought to you after I read up on linkerd and got excited about it being simpler, only to find this post saying Istio/App Mesh is the only game in Fargate town.

[mogul]

Cilium now supports a sidecar-free deployment mode using eBPF, and also provides ingress controller functionality! So we may want to switch to it instead of the ingress-nginx controller and Calico, and get secure east-west traffic to boot.
https://github.com/cilium/cilium/releases/tag/v1.12.0