Ingress doesn't listen on 443 if TLS cert not Valid

[markfermor]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):
TLS verify host ssl

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT or FEATURE REQUEST

NGINX Ingress controller version: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.4-gke.1", GitCommit:"04502ae78d522a3d410de3710e1550cfb16dad4a", GitTreeState:"clean", BuildDate:"2017-12-08T17:24:53Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}

Environment: testing

• Cloud provider or hardware configuration: GKE

• OS (e.g. from /etc/os-release):
  BUILD_ID=10032.50.0
  PRETTY_NAME="Container-Optimized OS from Google"
  VERSION=63

• Kernel (e.g. uname -a): Linux NODENAME 4.4.86+ Basic structure  #1 SMP Tue Nov 21 22:05:47 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.20GHz GenuineIntel GNU/Linux

• Install tools: GKE

• Others:

What happened:
Just tried setting up an nginx ingress (controller and Nginx running on K8s) which is also behind AWS cloudfront (for reasons I won't go into). So the layout looks something like this..
client (Requests x.hello.com) -> Cloudfront - terminates https (configured to send/forward to origin of y.goodbye.com and re-encrypt) -> K8s Nginx ingress - terminates https and has TLS for y.goodbye.com

When using this set-up there's different TLS Certificates involved one at Cloudfront level and another on the backend origin for Nginx. Problem is that Cloudfront doesn't re-write the Host/domain header of the request, so the request reaches the backend Nginx pods with host header of "x.hello.com" and Cloudfront is happy as long as the TLS cert served matches the origin of "y.goodbye.com".

However in order to get Nginx to answer with the right cert for a request coming into it of "x.hello.com" I configured the yaml as follows:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx-ext
  labels:
    allRelated: service
  name: ing-test
spec:
  rules:
  - host: x.hello.com
    http:
      paths:
      - backend:
          serviceName: service
          servicePort: 80
  tls:
  - hosts:
    - x.hello.com
    secretName: y.goodbye.com

This created nginx config of..

    ## start server x.hello.com
    server {
        server_name x.hello.com;

        listen 80;

        listen [::]:80;

        set $proxy_upstream_name "-";

        location / {
....

Which only listens on port 80 and not port 443. My guess is there's possibly some verification of the SSL which although is normally valid, in this case actually isn't valid.

What you expected to happen:
I expected the nginx config to listen on port 80 but also to listen on port 443 which is what does happen when the certificate is correctly matching the hosts array in tls config. I understand this is technically odd, but my hands are tied by Cloudfront and I needed Nginx to answer the SSL cert that Cloudfront is expecting to see.

How to reproduce it (as minimally and precisely as possible):
As described I suppose

Anything else we need to know:
I have a potential work around in that using the service-alias annotation possibly is enough to get the job done. However then it becomes more difficult I imagine (based on the layout of the nginx config and a hutch - i'm yet to test this theory), if I wanted to have multiple of the same server name to split traffic to different services based on request path, this wouldn't be possible. 1. I can't use it as a main server name again because i'm needing to use it as an alias to get around this problem. 2. Being part of an alias, i imagine if I make it part of an alias in another server {} definition else where that one will still gain priority over the other. Looks very similar to #1573 and this comment in particular #1573 (comment) - however in this case I need the wrong certificate to be returned (it's the correct certificate as far as Cloudfront is concerned but it's also technically the wrong certificate for the host - however I feel this is a perfectly valid use case?)

[aledbf]

Please post the logs from the nginx controller pod

[markfermor]

Just a small snippet - Thanks @aledbf

W0104 15:03:24.260457       8 controller.go:1062] unexpected error validating SSL certificate staging/y for host test.x.co.uk. Reason: x509: certificate is valid for *.y.io, y.io, not test.x.co.uk
W0104 15:03:24.264064       8 controller.go:1063] Validating certificate against DNS names. This will be deprecated in a future version.
W0104 15:03:24.264229       8 controller.go:1068] ssl certificate y does not contain a Common Name or Subject Alternative Name for host test.x.co.uk Reason: x509: certificate is valid for *.y.io, y.io, not test.x.co.uk
W0104 15:04:11.112723       8 controller.go:1062] unexpected error validating SSL certificate y for host test.x.co.uk Reason: x509: certificate is valid for *.y.io, y.io, not test.x.co.uk
W0104 15:04:11.113021       8 controller.go:1063] Validating certificate against DNS names. This will be deprecated in a future version.
W0104 15:04:11.113099       8 controller.go:1068] ssl certificate y does not contain a Common Name or Subject Alternative Name for host test.x.co.uk Reason: x509: certificate is valid for *.y.io, y.io, not test.x.co.uk
W0104 15:04:11.552635       8 controller.go:1062] unexpected error validating SSL certificate y for host test.x.co.uk Reason: x509: certificate is valid for *.y.io, y.io, not test.x.co.uk
W0104 15:04:11.552669       8 controller.go:1063] Validating certificate against DNS names. This will be deprecated in a future version.
W0104 15:04:11.552685       8 controller.go:1068] ssl certificate y does not contain a Common Name or Subject Alternative Name for host test.x.co.uk Reason: x509: certificate is valid for *.y.io, y.io, not test.x.co.uk
W0104 15:04:14.886266       8 controller.go:1062] unexpected error validating SSL certificate y for host test.x.co.uk Reason: x509: certificate is valid for *.y.io, y.io, not test.x.co.uk
W0104 15:04:14.886304       8 controller.go:1063] Validating certificate against DNS names. This will be deprecated in a future version.
W0104 15:04:14.886315       8 controller.go:1068] ssl certificate y does not contain a Common Name or Subject Alternative Name for host test.x.co.uk Reason: x509: certificate is valid for *.y.io, y.io, not test.x.co.uk

[aledbf]

@markfermor the log is clear, you have a secret with a SSL certificate for *.y.io, y.io but you are using test.x.co.uk in the tls section of the Ingress.

[markfermor]

Thanks @aledbf, and yes I understand that, however as I explained in the initial post, this is a valid situation that can occur when using AWS Cloudfront. It's valid to serve the domain with a different certificate to that of the originating requested Host header/domain. As described at the bottom of this doc (https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/using-https-cloudfront-to-custom-origin.html).

One of the domain names in the certificate must match one or both of the following values:

The value that you specified for Origin Domain Name for the applicable origin in your distribution.

If you configured CloudFront to forward the Host header to your origin, the value of the Host header. For more information about forwarding headers to your origin, see Configuring CloudFront to Cache Objects Based on Request Headers.

We have configured cloudfront to forward all headers, but technically speaking it's still valid for me to use a different certificate on the origin (Nginx Ingress) to terminate requests, to that of the certificate that cloudfront is using (why might i want to do this? - I can save costs of buying lots of SSL certs. I have lots of AWS ACM free SSL certs, but they have to stay within AWS and you can't export/take the certificate with you). However it's also valid with Cloudfront to just need the origin certificate to match the origin configured domain in cloudfront and not have to match the Host/domain of the request sent to CloudFront (thus technically we use this feature so we only need a single certificate external to cloudfront in order to validly serve requests)

The message also states "W0104 15:03:24.264064 8 controller.go:1063] Validating certificate against DNS names. This will be deprecated in a future version." Does that mean the verification is getting removed so that the usecase I'm trying to employ here will work?

[aledbf]

The message also states "W0104 15:03:24.264064 8 controller.go:1063] Validating certificate against DNS names. This will be deprecated in a future version." Does that mean the verification is getting removed so that the usecase I'm trying to employ here will work?

No. It means your certificate is not going to be valid. Using CN if SAN extension present is deprecated.
Please check this thread https://groups.google.com/a/chromium.org/forum/#!topic/security-dev/IGT2fLJrAeo

[aledbf]

Thanks @aledbf, and yes I understand that, however as I explained in the initial post, this is a valid situation that can occur when using AWS Cloudfront.......

This is not related to any external setup butto your local Ingress rule and secret containing the SSL certificate. Please check your secret contains a SSL certificate for the host defined in the ingress

[markfermor]

I'm not sure if you've fully understood what I'm trying to explain?
x.hello.com -> Cloudfront (cloudfront terminates SSL) -> Cloudfront is configured to send requests to backend (origin) of "y.goodbye.com" it will forward requests onto that endpoint and expects/accepts the connection as OK with that origin if the cert responded with is for a CN of "y.goodbye.com", when Cloudfront sends/forwards the request to that endpoint it's going directly to our Nginx Ingress, which see's original host header as "x.hello.com". So in order to configure this correctly so cloudfront see the SSL cert it's expecting to see, I need to be able to have x.hello.com from the ingress controller return the y.goodbye.com cert, which at the moment results in the listener for 443 being disabled (which I understand why that might have been done - but i'm saying it's not correct in this case, it's correct to be able to serve what's seemingly an invalid certificate, even though it's actually valid and what CloudFront is happy with)

[john-mcgowan-wowza]

@markfermor did you ever find a workaround for this issue? I'm running into a simiar situation. I want to associate a certificate with an Ingress but I don't need the hosts to match at this moment.

I've been looking through the nginx ingress controller annotations hoping that there is a "turn of certificate / host verification" option, but I'm not finding one.

[markfermor]

@john-mcgowan-wowza sort of yes. It's not ideal but it was a workable solution at the time. I configured the main server name to be something that was valid, and then added https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#server-alias a list of server-aliases that didn't match the certificate but would still be answered by the particular ingress resource, it was a space separated list of domain names (you can and we do add quite a few, but expect to have to tweak some of the Nginx memory settings if you are adding a large quantity of domains into the list). It worked like a charm

[markfermor]

#1873 (comment) no longer works. In 0.18.0 it worked like a charm, however just been working on upgrade to 0.25.0 and it doesn't work anymore. It's to do with moving SSL into LUA rather than Nginx handling it in the server blocks like it used to (for good intentions/reasons I might add!). To get around this we used --enable-dynamic-certificates=false, however as of 0.26.0 it's likely that flag will be removed and this will no longer be possible

[nimish]

Needed to use this issue at one point, but to clarify this: these rules only apply to web browsers, and not anything else that uses the internet. CAB guidelines are between CAs and Browsers -- if you deal with private CAs anything goes. There are definitely use cases that have not moved to SANs out there and the ingress controller should have a way to turn this bogus check off -- the browser will reject it if you use a malformed SAN-less cert anyway.

The message also states "W0104 15:03:24.264064 8 controller.go:1063] Validating certificate against DNS names. This will be deprecated in a future version." Does that mean the verification is getting removed so that the usecase I'm trying to employ here will work?

No. It means your certificate is not going to be valid. Using CN if SAN extension present is deprecated.
Please check this thread https://groups.google.com/a/chromium.org/forum/#!topic/security-dev/IGT2fLJrAeo

[TonyLovesDevOps]

Needed to use this issue at one point, but to clarify this: these rules only apply to web browsers, and not anything else that uses the internet. CAB guidelines are between CAs and Browsers -- if you deal with private CAs anything goes. There are definitely use cases that have not moved to SANs out there and the ingress controller should have a way to turn this bogus check off -- the browser will reject it if you use a malformed SAN-less cert anyway.

Indeed, especially in the banking industry where using private CAs to issue certs that browsers/default root CA stores see as "invalid" is a required practice. Even though we use ingress-nginx to great effect in many parts of our infrastructure, we're being forced to not use it for a similar use case because this check can't be disabled.

I love ingress-nginx and am grateful to its authors. It saves me tons of time and headaches in managing our infrastructure. So thanks! I'll keep using it wherever it works for us. That said, I too wish this validation check could be disabled so I could use it even more.