Default SSL Ciphers incompatible with AWS ELB

[jeffmhastings]

Is this a request for help? No

What keywords did you search in NGINX Ingress controller issues before filing this one? ELB

---

NGINX Ingress controller version: 0.10.2

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:51:54Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.8", GitCommit:"2f73858c9e6ede659d6828fe5a1862a48034a0fd", GitTreeState:"clean", BuildDate:"2018-02-09T21:23:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Centos 7
• Kernel (e.g. uname -a): Linux 3.10.0-693.17.1.el7.x86_64 Basic structure  #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kops
• Others:

What happened: ELB times out making SSL/HTTPS connection to the nginx ingress controller (NodePort).

What you expected to happen: ELB can terminate TLS and proxy to upstream nginx ingress controller port 443.

How to reproduce it (as minimally and precisely as possible): helm install the nginx-ingress chart using service type LoadBalancer and annotations:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: .............
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https

Anything else we need to know:

Nignx closes the TLS connection after the ClientHello. Trace from ssldump:

5 1  0.0006 (0.0006)  C>S  Handshake
      ClientHello
        Version 3.3
        cipher suites
        TLS_RSA_WITH_AES_256_GCM_SHA384
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
        TLS_RSA_WITH_AES_128_GCM_SHA256
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
        TLS_RSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
        TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
        TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
        TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA
        TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
        TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_EMPTY_RENEGOTIATION_INFO_SCSV
        compression methods
                  NULL
5 2  0.0007 (0.0001)  S>C  Alert
    level           fatal
    value           handshake_failure
5    0.0008 (0.0000)  S>C  TCP FIN
5    0.0012 (0.0003)  C>S  TCP FIN

As you can see, the ELB client has a limited set of supported ciphers. I was able to get this working in two ways:

1. Add an dhparam to the config and some DHE ciphers (e.x. DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384)
2. Add some less secure ciphers. I used the intermediate security option from https://mozilla.github.io/server-side-tls/ssl-config-generator/

I don't necessarily think this is a bug, but probably documentation should be added for getting ELBs to talk to the ingress controller over TLS.

[aledbf]

Closing. You can customize the ciphers in the configuration configmap.
Please check https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/tls.md#legacy-tls

[mentalblock]

I do agree that we need some documentation around this. The symptom of timeouts does not make it obvious where the problem is.

[adaphi]

It looks like this has fallen by the wayside?
Please can this go into the documentation somewhere! It's not at all obvious that this is the problem when it happens. Requests made to the ELB simply time out and are difficult to debug - they never reach the nginx instance, as it's the ELB's internal requests failing to make the handshake, not the request coming from outside.
It's such a simple, known fix, and so very close to the default deployment (and also I just spent two days tracking this down grmbl...)