Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IP Address SAN broken for TLS comms between nodes under 3.2.x #8206

Closed
robertglen opened this issue Jul 3, 2017 · 3 comments
Closed

IP Address SAN broken for TLS comms between nodes under 3.2.x #8206

robertglen opened this issue Jul 3, 2017 · 3 comments

Comments

@robertglen
Copy link

Bootstrapping a 3.2.1 etcd cluster was giving me endless errors like this:

Jul 03 21:54:45 ip-10-xxx-xxx-xxx etcd[3213]: rejected connection from “10.yyy.yyy.yyy:37194” (tls: “10.yyy.yyy.yyy” does not match any of DNSNames [“*.etcd.ec2.xyz” “etcd.ec2.xyz” “etcd0” “etcd0.ec2.xyz” “etcd1” “etcd1.ec2.xyz” “etcd2” “etcd2.ec2.xyz” “etcd3” “etcd3.ec2.xyz” “etcd4” “etcd4.ec2.xyz” “etcd5” “etcd5.ec2.xyz”])

It complains of the IP address 10.yyy.yyy.yyy not being amongst the DNSNames of the TLS cert provided by 10.yyy.yyy.yyy. Meanwhile 10.yyy.yyy.yyy x509 cert does have its address in the IP Address section of the cert's SAN.

This is new behavior. I simply swapped out the 3.2.1 etcd binary for the newest 3.1.x (3.1.9) and restarted etcd on all 5 nodes, now the cluster is fully operational.

@gyuho
Copy link
Contributor

gyuho commented Jul 5, 2017

“10.yyy.yyy.yyy” does not match any of DNSNames [“*.etcd.ec2.xyz” “etcd.ec2.xyz” “etcd0” “etcd0.ec2.xyz” “etcd1” “etcd1.ec2.xyz” “etcd2” “etcd2.ec2.xyz” “etcd3” “etcd3.ec2.xyz” “etcd4” “etcd4.ec2.xyz” “etcd5” “etcd5.ec2.xyz”])

Does any of the hosts resolve to 10.yyy.yyy.yyy?

@robertglen
Copy link
Author

No. I have NO DNS records matching any of these names. In my case, I have a script that generates an overloaded cert that could work for any of the hosts if I addressed them via DNS names, but I instead refer to all hosts by IP address in the etcd cmdline args. The script creates an x509 cert with all the names as you see here PLUS an IP Address SAN for 127.0.0.1 and the node's own IP address.

So each node technically has a unique cert with its own IP plus the collection of DNS SANs which as I said are not being used.

heyitsanthony pushed a commit to heyitsanthony/etcd that referenced this issue Jul 6, 2017
The IP SAN check would always do a DNS SAN check if DNS is given
and the connection's IP is verified. Instead, don't check DNS
entries if there's a matching iP.

Fixes etcd-io#8206
gyuho pushed a commit that referenced this issue Jul 7, 2017
The IP SAN check would always do a DNS SAN check if DNS is given
and the connection's IP is verified. Instead, don't check DNS
entries if there's a matching iP.

Fixes #8206
gyuho pushed a commit that referenced this issue Jul 7, 2017
The IP SAN check would always do a DNS SAN check if DNS is given
and the connection's IP is verified. Instead, don't check DNS
entries if there's a matching iP.

Fixes #8206
gyuho pushed a commit that referenced this issue Jul 7, 2017
The IP SAN check would always do a DNS SAN check if DNS is given
and the connection's IP is verified. Instead, don't check DNS
entries if there's a matching iP.

Fixes #8206
yudai pushed a commit to yudai/etcd that referenced this issue Oct 5, 2017
The IP SAN check would always do a DNS SAN check if DNS is given
and the connection's IP is verified. Instead, don't check DNS
entries if there's a matching iP.

Fixes etcd-io#8206
@ghost
Copy link

ghost commented Mar 21, 2019

Team, I have 3 masters of which one was manually deleted, and it was brought up by auto scale group however, etcd started to throw below on the new master that joined cluster. Any hint?

etcdmain: the server is already initialized as member before, starting as etcd member...
2019-03-21 16:32:25.541529 I | embed: peerTLS: cert = /srv/kubernetes/etcd.pem, key = /srv/kubernetes/etcd-key.pem, ca = , trusted-ca = /srv/kubernetes/ca.crt, client-cert-auth = true
2019-03-21 16:32:25.542119 I | embed: listening for peers on https://0.0.0.0:2380
2019-03-21 16:32:25.542161 I | embed: listening for client requests on 0.0.0.0:4001
2019-03-21 16:32:25.554167 I | pkg/netutil: resolving etcd-b.internal.test.com:2380 to 11.4.0.26:2380
2019-03-21 16:32:25.564718 I | pkg/netutil: resolving etcd-b.internal.test.com:2380 to 11.4.0.26:2380
2019-03-21 16:32:25.649568 I | etcdmain: rejected connection from "11.4.0.4:40736" (error "tls: "11.4.0.4" does not match any of DNSNames ["etcd-a.internal.test.com" "etcd-b.internal.test.com" "etcd-c.internal.test.com" "etcd-events-a.internal.test.com" "etcd-events-b.internal.test.com" "etcd-events-c.internal.test.com" "localhost"] (lookup etcd-events-c.internal.test.com on 100.64.0.10:53: dial udp 100.64.0.10:53: operation was canceled)", ServerName "etcd-b.internal.test.com", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a.internal.test.com" "etcd-b.internal.test.com" "etcd-c.internal.test.com" "etcd-events-a.internal.test.com" "etcd-events-b.internal.test.com" "etcd-events-c.internal.test.com" "localhost"])
2019-03-21 16:32:25.683192 C | etcdmain: member 16acea4929706d3 has already been bootstrapped

the etcd.pem shows below
X509v3 Subject Alternative Name:
DNS:etcd-a.internal.test.com, DNS:etcd-b.internal.test.com, DNS:etcd-c.internal.test.com, DNS:etcd-events-a.internal.test.com, DNS:etcd-events-b.internal.test.com, DNS:etcd-events-c.internal.test.com, DNS:localhost, IP Address:127.0.0.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants