Able to switch on/off SSL in Operator

[wenhaocs]

Operator is doing rolling restart on config update. But it does not help on SSL change. All 3 metads instances need to communicate with each other in order to be live. If only 1 restarts and changes from SSL on to SSL off, it is unable to get heartbeats from other instances. But operator will not continue to restart the next metad instance if the prior restart has not finished...

E20240522 17:48:39.582396    77 PeekingManager.h:115] Received SSL connection on non SSL port
E20240522 17:48:39.586596    78 PeekingManager.h:115] Received SSL connection on non SSL port
E20240522 17:48:39.590667    79 PeekingManager.h:115] Received SSL connection on non SSL port
E20240522 17:48:39.594632    80 PeekingManager.h:115] Received SSL connection on non SSL port
E20240522 17:48:39.598673    65 PeekingManager.h:115] Received SSL connection on non SSL port
E20240522 17:48:39.602654    66 PeekingManager.h:115] Received SSL connection on non SSL port

mau-comm-metad-0                     3/3     Running   0          7d20h
mau-comm-metad-1                     3/3     Running   0          7d20h
mau-comm-metad-2                     2/3     Running   0          12m

rkovvuri@HKWVMPM23L graph-service % kubectl describe pod mau-comm-metad-2 -n mau-comm
Normal   Started    13m                  kubelet           Started container statsd-pubsub
  Warning  Unhealthy  3m1s (x68 over 12m)  kubelet           Readiness probe failed: Get "http://10.66.94.221:19559/status": dial tcp 10.66.94.221:19559: connect: connection refused

[MegaByte875]

The testing cases all passed with operator v1.8.0.

• SSL on to SSL off
• SSL off to SSL on

Please note that internal RPC interface calls will fail during SSL switchover.

[wenhaocs]

Please prioritize this for internal RPC interface. E.g., do not transfer leaders for SSL config changes.