Update Golang to 1.15.7 (#22495) (#23630)

This PR updates the Go version in the repository to 1.15.7.

Numerous improvements are coming with the new version of Golang. The most noteworthy is the binary sizes are reduced by 5-8%. You can check out the [release notes of Golang 1.15](https://golang.org/doc/go1.15) for more information.

Another change is that the deprecated, legacy behaviour of treating the CommonName field on X.509 certificates as a hostname when no Subject Alternative Names are present is now disabled by default. We do not expect users to update their certificates immediately. Thus, the default behaviour is kept for all existing verification modes. Furthermore, a new verification mode is added named `strict`. If this mode is selected Beats expects to have a SAN in the certificate.

`verification_mode` | behaviour
------- | -----
`""` | CommonName is checked if no DNSName is available, then full check (previous behaviour)
`"strict"` | full check with new strict SAN requirements
`"full"` | CommonName is checked if no DNSName is available, then full check (previous behaviour, default value)
`"certificate"` | certificate check without hostname (previous behaviour)
`"none"` | no check (previous behaviour)

The legacy behaviour is going to be removed in the next major version 8.0. We expect users to update their certificates so it does not depend on CommonName.

(cherry picked from commit e4ef5cc)

.go-version

@@ -1 +1 @@
-1.14.12
+1.15.7

CHANGELOG-developer.next.asciidoc

@@ -98,3 +98,4 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
 - Make the mage binary used by the build process in the docker container to be statically compiled. {pull}20827[20827]
 - Add support for customized monitoring API. {pull}22605[22605]
 
+- Update Go version to 1.15.7. {pull}22495[22495]

CHANGELOG.next.asciidoc

@@ -691,6 +691,11 @@ port. {pull}19209[19209]
 
 *Affecting all Beats*
 
+- Selecting `full` in `ssl.verification_mode` option will not treat CommonName field in x509 certificates as
+  a hostname when Subject Alternative Name is not present from v8.0.
+  Please update your certificates so it contains at least one DNSName instead of relying on CommonName in the new
+  major version of Beats.
+
 *Filebeat*
 
 - The experimental modules for Citrix Netscaler and Symantec Endpoint Protection have been removed.

auditbeat/Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.14.12
+FROM golang:1.15.7
 
 RUN \
     apt-get update \

auditbeat/auditbeat.reference.yml

@@ -530,6 +530,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -657,6 +663,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -854,6 +866,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -1010,6 +1028,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -1302,6 +1326,12 @@ setup.kibana:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -1499,6 +1529,12 @@ logging.files:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary

filebeat/Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.14.12
+FROM golang:1.15.7
 
 RUN \
     apt-get update \

filebeat/filebeat.reference.yml

@@ -1409,6 +1409,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -1536,6 +1542,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -1733,6 +1745,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -1889,6 +1907,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -2181,6 +2205,12 @@ setup.kibana:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -2378,6 +2408,12 @@ logging.files:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary

filebeat/input/kafka/config.go

@@ -177,7 +177,7 @@ func newSaramaConfig(config kafkaInputConfig) (*sarama.Config, error) {
 	}
 	if tls != nil {
 		k.Net.TLS.Enable = true
-		k.Net.TLS.Config = tls.BuildModuleConfig("")
+		k.Net.TLS.Config = tls.BuildModuleClientConfig("")
 	}
 
 	if config.Kerberos.IsEnabled() {

filebeat/input/mqtt/client.go

@@ -40,7 +40,7 @@ func createClientOptions(config mqttInputConfig, onConnectHandler func(client li
 		if err != nil {
 			return nil, err
 		}
-		clientOptions.SetTLSConfig(tlsConfig.BuildModuleConfig(""))
+		clientOptions.SetTLSConfig(tlsConfig.BuildModuleClientConfig(""))
 	}
 	return clientOptions, nil
 }

filebeat/inputsource/tcp/server.go

@@ -68,7 +68,7 @@ func (s *Server) createServer() (net.Listener, error) {
 	var l net.Listener
 	var err error
 	if s.tlsConfig != nil {
-		t := s.tlsConfig.BuildModuleConfig(s.config.Host)
+		t := s.tlsConfig.BuildServerConfig(s.config.Host)
 		l, err = tls.Listen("tcp", s.config.Host, t)
 		if err != nil {
 			return nil, err

filebeat/tests/system/test_tcp_tls.py

@@ -127,6 +127,8 @@ def test_tcp_over_tls_and_verify_invalid_server_without_mutual_auth(self):
         with pytest.raises(ssl.SSLError):
             tls.connect((config.get('host'), config.get('port')))
 
+        sock.close()
+
     def test_tcp_over_tls_mutual_auth_fails(self):
         """
         Test filebeat TCP with TLS with default setting to enforce client auth, with bad client certificates
@@ -171,6 +173,8 @@ def test_tcp_over_tls_mutual_auth_fails(self):
             # so that the failure can be reported as an exception when it arrives.
             tls.recv(1)
 
+        sock.close()
+
     def test_tcp_over_tls_mutual_auth_succeed(self):
         """
         Test filebeat TCP with TLS when enforcing client auth with good client certificates.
@@ -275,6 +279,8 @@ def test_tcp_tls_with_a_plain_text_socket(self):
 
         assert path.isfile(path.join(self.working_dir, "output/" + self.beat_name)) is False
 
+        sock.close()
+
     def assert_output(self, output):
         assert len(output) == 2
         assert output[0]["input.type"] == "tcp"

go.mod

@@ -1,6 +1,6 @@
 module github.com/elastic/beats/v7
 
-go 1.14
+go 1.15
 
 require (
 	4d63.com/tz v1.1.1-0.20191124060701-6d37baae851b

heartbeat/Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.14.12
+FROM golang:1.15.7
 
 RUN \
     apt-get update \

heartbeat/heartbeat.reference.yml

@@ -707,6 +707,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -834,6 +840,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -1031,6 +1043,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -1187,6 +1205,12 @@ output.elasticsearch:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -1479,6 +1503,12 @@ setup.kibana:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary
@@ -1676,6 +1706,12 @@ logging.files:
   # matches the names identified within the certificate.
   # * certificate, which verifies that the provided certificate is signed by a
   # trusted authority (CA), but does not perform any hostname verification.
+  # * strict, which verifies that the provided certificate is signed by a trusted
+  # authority (CA) and also verifies that the server's hostname (or IP address)
+  # matches the names identified within the certificate. If the Subject Alternative
+  # Name is empty, it returns an error.
+  # * certificate, which verifies that the provided certificate is signed by a
+  # trusted authority (CA), but does not perform any hostname verification.
   #  * none, which performs no verification of the server's certificate. This
   # mode disables many of the security benefits of SSL/TLS and should only be used
   # after very careful consideration. It is primarily intended as a temporary