GLBC: k8s 1.6 fails to replace certificate

[bbzg]

Environment: Kubernetes 1.6 on GKE.

On https://console.cloud.google.com/home/activity?project=MyProjectName I saw that every 10 minutes kubernetes tried to replace the SSL certificate on GLBC, with the following error message:

User              [email protected]
Resource name     projects/MyProjectName/global/sslCertificates/k8s-ssl-1-default-ingress-xxxxxxxxx
Error message     409: The resource 'projects/MyProjectName/global/sslCertificates/k8s-ssl-1-default-ingress-xxxxxxxxx' already exists
Request
  Certificate     REDACTED
  Name            k8s-ssl-1-default-ingress-xxxxxxxxx
  Private key     REDACTED
Response
  Error
    Code          409
    Errors
      Errors 1
        Domain    global
        Message   The resource 'projects/MyProjectName/global/sslCertificates/k8s-ssl-1-default-ingress-xxxxxxxxx' already exists
        Reason    alreadyExists
      Message     The resource 'projects/MyProjectName/global/sslCertificates/k8s-ssl-1-default-ingress-xxxxxxxxx' already exists

I then proceeded to go to https://console.cloud.google.com/networking/loadbalancing/advanced/sslCertificates/list?project=MyProjectName and removed the unused certificate k8s-ssl-default-ingress-xxxxxxxxx (notice that this was the unused cert but did not have the ssl-1 name).

After a short while there was a new ssl-default cert (same suffix as before) marked as used, and the ssl-1-default cert was marked as unused.

Now, every 10 minutes https://console.cloud.google.com/home/activity?project=MyProjectName reports that the SSL cert on GLBC was successfully updated.

The state of my certs are now
ssl-1-default: Unused.
ssl-default: Used.
No error messages in the activity log.

[nicksardo]

I'm able to reproduce with the following steps;

1. Update a secret so an existing ingress now uses a certificate named k8s-ssl-1-..... (let's call this the "secondary name")
2. Restart the ingress controller

The following is logged:

I0419 23:38:11.398936       5 loadbalancers.go:122] Creating l7 default-my-echo-ingress-https--553ddaa9133fca24
I0419 23:38:11.609334       5 loadbalancers.go:410] Creating new sslCertificates k8s-ssl-default-my-echo-ingress-https--553ddaa9133fca24 for default-my-echo-ingress-https--553ddaa9133fca24
I0419 23:38:15.217445       5 loadbalancers.go:456] Https proxy k8s-tps-default-my-echo-ingress-https--553ddaa9133fca24 has the wrong ssl certs, setting https://www.googleapis.com/compute/v1/projects/nicksardo-playground/global/sslCertificates/k8s-ssl-default-my-echo-ingress-https--553ddaa9133fca24 overwriting https://www.googleapis.com/compute/v1/projects/nicksardo-playground/global/sslCertificates/k8s-ssl-1-default-my-echo-ingress-https--553ddaa9133fca24

During the first sync after restart, the certificate is re-created using the primary name, the HTTPS Target Proxy is updated, but the certificate with the secondary name is not deleted. On next secret change, the controller will fail to create a certificate since the secondary name is used.

I'll write up a fix.

[nicksardo]

cc'ing @thockin who has experienced this before.

[jmhodges]

I also have a cluster this has been happening in for many months. If y'all need more data, I'm willing to provide!