TLS protocol version issue

[jmg011]

Started seeing harvest failures on 7mode filers. Any idea about this issue? Is it TLS version issue?

I'm not aware of any TLS update on 7mode systems.

{"level":"warn","Poller":"netappfiler","caller":"./poller.go:660","time":"2022-05-05T18:11:48-07:00","message":"init collector-object (Zapi:Node): connection error => connection error => Post "https://netappfilerFQDN:443/servlets/netapp.servlets.admin.XMLrequest_filer\": tls: server selected unsupported protocol version 301"}

[rahulguptajss]

@jmg011 See if issue #540 helps?

[Hardikl]

@jmg011
Any recent changes for auth_style and use_insecure_tls fields in harvest.yaml for this 7mode filer ?

[jmg011]

@Hardikl @rahulguptajss Hey, no we did not make any changes. There are so many 7mode filers having this issue. I debugged this further to check the timeline on when this stopped working. It aligns with the exact time I upgraded Harvest to the latest version. Is there any change in new Harvest version that could pose such issue?

[cgrinds]

hi @jmg011 we're not aware of any changes in this area. We can do a git bisect if you let us know which version worked and what version doesn't.

• Did you upgrade to nightly or 22.02?

I double checked and nightly has no problems monitoring 7-mode filers.

• Are you using use_insecure_tls: true in both cases?
• Are these filers using Vault and if so any changes there?

What if we remove Harvest from the loop altogether and use a recent version of curl to verify filer connectivity and auth?

Can you try this, replace username/pass and ip?

curl --version
curl 7.83.0 (x86_64-apple-darwin21.3.0) libcurl/7.83.0 (SecureTransport) OpenSSL/1.1.1n zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.2 libssh2/1.10.0 nghttp2/1.47.0 librtmp/2.3 OpenLDAP/2.6.1
Release-Date: 2022-04-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets zstd

curl --insecure --verbose --include --user user:pass --data '<?xml version="1.0" encoding="UTF-8"?> <netapp xmlns="http://www.netapp.com/filer/admin" version="1.21"> <system-get-version/> </netapp>' -H "Content-Type: text/xml" https://10.65.51.134/servlets/netapp.servlets.admin.XMLrequest_filer

output

*   Trying 10.65.51.134:443...
* Connected to 10.65.51.134 (10.65.51.134) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=fas2240-2.; C=US; ST=California; L=Santa Clara; O=Your Company
*  start date: May 26 17:01:28 2020 GMT
*  expire date: May 23 17:01:28 2035 GMT
*  issuer: CN=fas2240-2.; C=US; ST=California; L=Santa Clara; O=Your Company
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Server auth using Basic with user 'root'
> POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1
> Host: 10.65.51.134
> Authorization: Basic cm9vdbcdNGlwdjY=
> User-Agent: curl/7.83.0
> Accept: */*
> Content-Type: text/xml
> Content-Length: 136
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 Ok
HTTP/1.1 200 Ok
< MIME-Version: 1.1
MIME-Version: 1.1
< Server: NetApp//8.2.5P5
Server: NetApp//8.2.5P5
< Date: Mon, 09 May 2022 12:38:14 GMT
Date: Mon, 09 May 2022 12:38:14 GMT
< Content-Type: text/xml
Content-Type: text/xml
< Connection: Close
Connection: Close

< 
<?xml version='1.0' encoding='UTF-8' ?>
<!DOCTYPE netapp SYSTEM '/na_admin/netapp_filer.dtd'>
<netapp version='1.1' xmlns='http://www.netapp.com/filer/admin'>
* TLSv1.2 (IN), TLS alert, close notify (256):
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
<results status="passed"><version>NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021</version><is-clustered>false</is-clustered></results></netapp>⏎   

[jmg011]

@cgrinds

Current Harvest Version:

bin/harvest --version
harvest version 22.03.3118-nightly (commit cf0ed8d)

This is set to true

use_insecure_tls: true

7mode filers are not using vault and no changes were made to filers. I noticed 7-mode filers with NetApp Release 8.2P4 or Less does not work with this version of Harvest. Rest of the 7mode work fine. That is the case with our environment.

CURL output:

curl --insecure --verbose --include --user 'myUSER@myPASSWORD' --data '<?xml version="1.0" encoding="UTF-8"?> <netapp xmlns="http://www.netapp.com/filer/admin" version="1.21"> <system-get-version/> </netapp>' -H "Content-Type: text/xml" https://MYipADDRESS/servlets/netapp.servlets.admin.XMLrequest_filer
* About to connect() to MYipADDRESS port 443 (#0)
*   Trying MYipADDRESS...
* Connected to MYipADDRESS (MYipADDRESS) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -12156 (SSL_ERROR_WEAK_SERVER_CERT_KEY)
* The server certificate included a public key that was too weak.
* Closing connection 0
curl: (35) The server certificate included a public key that was too weak.

[cgrinds]

thanks @jmg011 that's very helpful. @rahulguptajss found this Go change log note that we're confident is effecting you. It would be best if you upgraded your ONTAP versions to support TLS 1.2+ since the whole industry is trying to phase out TLS 1.0.

We'll take a look at adding a TLS minVersion flag to Harvest. In the meantime, can you try the env var workaround?

If Config.MinVersion is not set, it now defaults to TLS 1.2 for client connections. Any safely up-to-date server is expected to support TLS 1.2, and browsers have required it since 2020. TLS 1.0 and 1.1 are still supported by setting Config.MinVersion to VersionTLS10. The server-side default is unchanged at TLS 1.0.

The default can be temporarily reverted to TLS 1.0 by setting the GODEBUG=tls10default=1 environment variable. This option will be removed in Go 1.19.

[cgrinds]

Fixed in #1017

[jmg011]

@cgrinds @rahulguptajss @Hardikl - I could fix it with the workaround. Thank you for your quick response on this and updating the next release to support this old version. WIP to get rid of those systems so hopefully in the future releases we won't face the same issue.

[rahulguptajss]

@jmg011 Thanks for letting us know that workaround works. old tls version support is added for next release #1017

[jmg011]

Hi @cgrinds @rahulguptajss

This issue is returned with the upgrade to the 22.11.0-1 version.

I set this env variable but did not help!

export GODEBUG=tls10default=1

2023-01-25T18:21:06Z DBG ./poller.go:1130 > Failed to upgrade to Rest. Use collector error="connection error Get "https:///api/cluster?return_records=true&fields=*": tls: server selected unsupported protocol version 301" Poller= collector=Zapi
2023-01-25T18:21:06Z DBG ./poller.go:618 > Merged template. Poller= template=custom.yaml
2023-01-25T18:21:06Z DBG ./poller.go:669 > Starting collectors Poller= collectors=1
2023-01-25T18:21:06Z DBG zapi/client.go:581 > Using timeout Poller= Zapi=Client timeout=30s
2023-01-25T18:21:08Z WRN ./poller.go:678 > abort collector Poller= collector=Zapi object=Aggregate
2023-01-25T18:21:08Z WRN ./poller.go:320 > no collectors initialized, stopping Poller=
2023-01-25T18:21:08Z INF ./poller.go:545 > cleaning up and stopping [pid=4979] Poller=

Any idea what I can do to fix this issue quickly?

[cgrinds]

hi @jmg011 is this happening for your 7-mode filers or cdot ones?

Try editing your harvest.yml file and adding a key for tls_min_version to the poller section like this:

Pollers:
    mycluster:
        datacenter: dc-1
        addr: 10.X.X.X
        auth_style: basic_auth
        tls_min_version: tls10   # <=== see https://netapp.github.io/harvest/22.11/configure-harvest-basic/#pollers

[jmg011]

@cgrinds - Thank you. This happened for 7mode clusters.

I applied the fix you suggested and it worked.

Environment variable stopped working though, just fyi.

[cgrinds]

Awesome, glad that worked.

Go removed the GODEBUG=tls10default=1 environment variable in Go 1.19. Thankfully, you raised this issue last year and we were able to add the tls_min_version option in 22-05.

Harvest 22-05 was compiled with Go 1.18 which included the envvar, but Harvest 22.11 is compiled with Go 1.19 which removed it.