Etcd Ignoring x509 IP SANs Value

[PatrickMurray]

etcd ignoring x509 IP SANS field

Good morning all,

I'm attempting to secure a five node cluster using my organization's internal certificate authority. To provide some background, we're using Ansible to automate node configuration management. Previously, we've been successful in deploying clusters which communicate over plaintext transmission.

At this point in time, we've issued signed certificates for each machine in the cluster and have specified the machine's IP address as the subject alternative name, per x509; however, etcd is complaining stating that the field is missing.

Any guidance in resolving this issue would greatly be appreciated.

journalctl -xe

Jul 27 14:13:15 ip-XXX-XXX-XXX-XXX etcd[8659]: 7d96a2caa2c87b1f became candidate at term 704
Jul 27 14:13:15 ip-XXX-XXX-XXX-XXX etcd[8659]: 7d96a2caa2c87b1f received MsgVoteResp from 7d96a2caa2c87b1f at term 704
Jul 27 14:13:15 ip-XXX-XXX-XXX-XXX etcd[8659]: 7d96a2caa2c87b1f [logterm: 1, index: 5] sent MsgVote request to f5da8012c0ba50c8 at term 704
Jul 27 14:13:15 ip-XXX-XXX-XXX-XXX etcd[8659]: 7d96a2caa2c87b1f [logterm: 1, index: 5] sent MsgVote request to 658b309332a5287a at term 704
Jul 27 14:13:15 ip-XXX-XXX-XXX-XXX etcd[8659]: 7d96a2caa2c87b1f [logterm: 1, index: 5] sent MsgVote request to ba2dc4fffc7fc92a at term 704
Jul 27 14:13:15 ip-XXX-XXX-XXX-XXX etcd[8659]: 7d96a2caa2c87b1f [logterm: 1, index: 5] sent MsgVote request to c5b43a4baf6a38cd at term 704
Jul 27 14:13:16 ip-XXX-XXX-XXX-XXX etcd[8659]: publish error: etcdserver: request timed out
Jul 27 14:13:16 ip-XXX-XXX-XXX-XXX etcd[8659]: health check for peer 658b309332a5287a could not connect: x509: cannot validate certificate for XXX-XXX-XXX-XXX because it doesn't contain any IP SANs
Jul 27 14:13:16 ip-XXX-XXX-XXX-XXX etcd[8659]: health check for peer ba2dc4fffc7fc92a could not connect: x509: cannot validate certificate for XXX-XXX-XXX-XXX because it doesn't contain any IP SANs
Jul 27 14:13:16 ip-XXX-XXX-XXX-XXX etcd[8659]: health check for peer c5b43a4baf6a38cd could not connect: x509: cannot validate certificate for XXX-XXX-XXX-XXX because it doesn't contain any IP SANs
Jul 27 14:13:16 ip-XXX-XXX-XXX-XXX etcd[8659]: health check for peer f5da8012c0ba50c8 could not connect: x509: cannot validate certificate for XXX-XXX-XXX-XXX because it doesn't contain any IP SANs
Jul 27 14:13:17 ip-XXX-XXX-XXX-XXX etcd[8659]: 7d96a2caa2c87b1f is starting a new election at term 704

systemd.unit

/etcd/systemd/system/etcd_cluster.service

[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service


[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

                                                                                                                                                              \
ExecStart=/usr/bin/etcd                                                                                                                                       \
  --name                        etcd_node_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX                                     \
  --data-dir                    /var/lib/etcd                                                                                                                 \
                                                                                                                                                              \
  --listen-client-urls          https://XXX.XXX.XXX.XXX:2379,https://127.0.0.1:2379  \
  --advertise-client-urls       https://XXX.XXX.XXX.XXX:2379                                                         \
                                                                                                                                                              \
  --listen-peer-urls            https://XXX.XXX.XXX.XXX:2380                                                           \
  --initial-advertise-peer-urls https://XXX.XXX.XXX.XXX:2380                                                           \
                                                                                                                                                              \
  --client-cert-auth                                                                                                                                          \
  --ca-file                     /opt/easyrsa/pki/ca.crt                                                                                                       \
  --trusted-ca-file             /opt/easyrsa/pki/ca.crt                                                                                                       \
  --cert-file                   /opt/easyrsa/pki/XXX.XXX.XXX.XXX.crt                                                                                 \
  --key-file                    /opt/easyrsa/pki/private/XXX.XXX.XXX.XXX.key                                                                         \
                                                                                                                                                              \
  --peer-client-cert-auth                                                                                                                                     \
  --peer-ca-file                /opt/easyrsa/pki/ca.crt                                                                                                       \
  --peer-trusted-ca-file        /opt/easyrsa/pki/ca.crt                                                                                                       \
  --peer-cert-file              /opt/easyrsa/pki/XXX.XXX.XXX.XXX.crt                                                                                 \
  --peer-key-file               /opt/easyrsa/pki/private/XXX.XXX.XXX.XXX.key                                                                         \
                                                                                                                                                              \
  --initial-cluster             etcd_node_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=https://XXX.XXX.XXX.XXX:2380,etcd_node_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=https://XXX.XXX.XXX.XXX:2380,etcd_node_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=https://XXX.XXX.XXX.XXX:2380,etcd_node_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=https://XXX.XXX.XXX.XXX:2380,etcd_node_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=https://XXX.XXX.XXX.XXX:2380                                                                                                                                                  \
  --initial-cluster-token       mytoken                                                                                                                       \
  --initial-cluster-state       new                                                                                                                           \
                                                                                                                                                              \
  --auto-compaction-retention   1


[Install]
WantedBy=multi-user.target

x509 Certificate

/opt/easyrsa/pki/ca.crt

-----BEGIN CERTIFICATE-----
MIIEuTCCA6GgAwIBAgIJAKCiHaLjPbOEMA0GCSqGSIb3DQEBDQUAMIGUMQswCQYD
...
m/LbPQessopRWecBYQ==
-----END CERTIFICATE-----

/opt/easyrsa/pki/XXX.XXX.XXX.XXX.crt

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 3 (0x3)
    Signature Algorithm: sha512WithRSAEncryption
        Issuer: C=US, ST=New Jersey, L=Matawan, O=iCIMS, Inc, OU=DevOps, CN=XXX.XXX.XXX.XXX/[email protected]
        Validity
            Not Before: Jul 27 13:46:08 2017 GMT
            Not After : Jul 25 13:46:08 2027 GMT
        Subject: C=US, ST=New Jersey, L=Matawan, O=iCIMS, Inc, OU=DevOps, CN=XXX.XXX.XXX.XXX/[email protected]
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ab:84:21:e1:9a:0e:36:8c:4e:85:48:e2:c3:38:
                    ...
                    bc:99
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Subject Key Identifier: 
                0E:FB:01:A2:C1:7B:EA:E7:4A:71:BB:80:AC:6E:4C:39:A0:C5:EA:C1
            X509v3 Authority Key Identifier: 
                keyid:F5:D3:F5:E5:7A:F8:34:70:41:18:8F:59:4B:2C:92:34:EE:E9:01:50
                DirName:/C=US/ST=New Jersey/L=Matawan/O=iCIMS, Inc/OU=DevOps/CN=XXX.XXX.XXX.XXX/[email protected]
                serial:A0:A2:1D:A2:E3:3D:B3:84

            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
            X509v3 Subject Alternative Name: 
                IP Address:XXX.XXX.XXX.XXX
    Signature Algorithm: sha512WithRSAEncryption
         43:fc:70:78:4a:9c:db:d2:b2:6b:01:cd:bb:81:f9:b5:08:77:
         ...
         ce:65:70:15
-----BEGIN CERTIFICATE-----
MIIE5TCCA82gAwIBAgIBAzANBgkqhkiG9w0BAQ0FADCBlDELMAkGA1UEBhMCVVMx
...
Su9+qO7OZXAV
-----END CERTIFICATE-----

/opt/easyrsa/pki/private/XXX.XXX.XXX.XXX.key

-----BEGIN PRIVATE KEY-----
Not today! :)
-----END PRIVATE KEY-----

[gyuho]

What etcd version are you running?

[PatrickMurray]

@gyuho Oops, totally forgot!

etcdctl version: 3.1.9
API version: 2

[gyuho]

Seems like a same issue with #8268?
The fix #8281 will be included in our next patch release, this or next week.

[PatrickMurray]

@gyuho I'm not familiar with the particulars of that issue - although upon a quick glance, that ticket appears to regard the DNS SANs, not IP SANs. I was able to resolve my problem by setting the SAN as IP=XXX.XXX.XXX.XXX,DNS=XXX.XXX.XXX.XXX. In my opinion, this problem seems to originate from golang's openssl implementation of SANs. Thanks for the help!

[vhosakot]

I saw the same error when using mysql client in golang:

Failed to connect to database:  x509: cannot validate certificate for 10.111.202.229 because it doesn't contain any IP SANs

and setting InsecureSkipVerify to true (to skip verification of certificate) resolved it for me:

https://godoc.org/crypto/tls#Config