HA Deployment of OM does not work on Kubernetes when listens on external IP

[dantalian-pv]

By following instruction on the documentation page https://ozone.apache.org/docs/1.3.0/feature/om-ha.html
I am trying to make a HA deployment for Apache Ozone, including replication for OM.
Deployment with single OM node is working well, but one problem appears with enabled ratis and replication .

When single OM node is deployed, OM takes into account ozone.om.address = 0.0.0.0:9862, and then listens on wildcard IP. With that Kubernetes is able to redirect traffic from headless service to a running application in a Pod's container.

Command executed inside the container:

$> ss -nltp
State       Recv-Q Send-Q    Local Address:Port      Peer Address:Port              
LISTEN      0      128                   *:9874      *:*                   users:(("java",pid=7,fd=233))
LISTEN      0      256                   *:9862      *:*                   users:(("java",pid=7,fd=218))
LISTEN      0      4096                  *:9872      *:*                   users:(("java",pid=7,fd=231))

However, when ratis is enabled and service.ids are defined, OM on bootstrap takes external IP as listen address, and because of that Kubernetes is not able to redirect traffic though the headless service to the application in the Pod's container anymore.

Command executed inside the container:

$> ss -nltp
State       Recv-Q Send-Q       Local Address:Port          Peer Address:Port              
LISTEN      160    256          10.42.5.225:9862            *:*                   users:(("java",pid=6,fd=213))

Is it possible to configure OM somehow, so that it will always listen on wildcard address, but will use configured node addresses to find other replication nodes?

P.S. Here is enclosed Kubernetes configuration and logs:
om_1.zip

[kerneltime]

cc @szetszwo

[GeorgeJahad]

@mladjan-gadzic Please take a look.

[GeorgeJahad]

@sokui I know you have quite a bit of experience with ozone on k8s. Have you run into this problem?

[sokui]

Hi @GeorgeJahad ,

I have set up ozone in k8s with kerberos and HA enabled. I noticed that I do not have this line in my config. Not sure if it is the root cause for your case.

  OZONE-SITE.XML_ozone.om.address: 0.0.0.0:9862

[dantalian-pv]

Hi @sokui ,
could you check on which IP address OM service is listening inside the docker container on Kubernetes, when it's deployed with HA enabled?
For that I have to make a custom docker image:
Dockerfile.zip

Then inside the container you can execute ss -nltp.

[sokui]

Hi @dantalian-pv ,

I tried to use ss in my current deployed pod, but the tool is not available. I will check later today when I got chance. My ozone version may be different from yours, and also I have some other scripts in my docker. So I cannot directly use your docker image. But I will figure it out to install the tool. Will get back to you today or tomorrow.

[sokui]

Hi @dantalian-pv , sorry for late reply. Here is my working om ss -nltp results:

$ ss -nltp
State      Recv-Q Send-Q                                                              Local Address:Port                                                                             Peer Address:Port
LISTEN     0      0                                                         [::ffff:100.116.88.103]:9862                                                                                     [::]:*                   users:(("java",pid=75,fd=239))
LISTEN     0      0                                                                            [::]:9872                                                                                     [::]:*                   users:(("java",pid=75,fd=249))
LISTEN     0      0                                                                            [::]:9874                                                                                     [::]:*                   users:(("java",pid=75,fd=251))

[sokui]

One thing you should be aware is that in k8s, when a pod just gets started, it takes some time to be listed behind a service. This means the FQDN ozz-ozone-om-0.ozz-ozone-om.test-namespace.svc.cluster.local is not resolvable initially. I am not sure if your problem is this issue.

[dantalian-pv]

Hi @sokui,
thank you for your reply. My first question is whether you have a --bootstrap flag in your OM container configuration or not?

I think I have multiple issues at the same time, however one can be the reason for other.

First I did some further research and noticed, that when an RPC server starts listening on port 9862 it does not reply on this port to any request, event inside the container to any other instance or to local instance running in the container (I've tested it with netcat). To investigate this problem I have made thread dumps from 2 instances, which you can find at the attachment:
om_HA-threaddumps.zip

It looks like it hangs when it is trying to connect to other nodes, but because all of them do not reply via RPC, they hang forever at this point. You can see it because neither UI nor other listeners has started, as shown at my first message (please, see top of this thread).

However I think my issue is related to the point that the RPC server in a container listens on external IP address instead of listening on wildcard, and because of specific configuration of my Kubernetes cluster (which I cannot change), the Kubernetes cannot redirect traffic to the RPC server inside the container. With that I am asking whether it is possible to configure RPC of OM to listen on wildcard address, but taking into account remote IP addresses of other instances, configured for HA, to connect to them.

In comparison here is a part of ConfigMap for HA of SCM and output of ss -nltp, which works for me:

OZONE-SITE.XML_ozone.scm.address.ozz-ozone-scm.ozz-ozone-scm-0: ozz-ozone-scm-0.ozz-ozone-scm.test-namespace.svc.cluster.local
  OZONE-SITE.XML_ozone.scm.address.ozz-ozone-scm.ozz-ozone-scm-1: ozz-ozone-scm-1.ozz-ozone-scm.test-namespace.svc.cluster.local
  OZONE-SITE.XML_ozone.scm.address.ozz-ozone-scm.ozz-ozone-scm-2: ozz-ozone-scm-2.ozz-ozone-scm.test-namespace.svc.cluster.local
  OZONE-SITE.XML_ozone.scm.datanode.id.dir: /data
  OZONE-SITE.XML_ozone.scm.nodes.ozz-ozone-scm: ozz-ozone-scm-0,ozz-ozone-scm-1,ozz-ozone-scm-2,
  OZONE-SITE.XML_ozone.scm.primordial.node.id: ozz-ozone-scm-0
  OZONE-SITE.XML_ozone.scm.ratis.enable: "true"
  OZONE-SITE.XML_ozone.scm.service.ids: ozz-ozone-scm

bash-4.2$ ss -nltp
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128          *:9876                     *:*                   users:(("java",pid=7,fd=215))
LISTEN     0      256          *:9860                     *:*                   users:(("java",pid=7,fd=202))
LISTEN     0      256          *:9861                     *:*                   users:(("java",pid=7,fd=181))
LISTEN     0      4096         *:9894                     *:*                   users:(("java",pid=7,fd=214))
LISTEN     0      4096         *:9895                     *:*                   users:(("java",pid=7,fd=216))
LISTEN     0      256          *:9863                     *:*                   users:(("java",pid=7,fd=192))

[sokui]

Hi @dantalian-pv ,

I start om using ozone om --init && ozone om. Can you try that?

[dantalian-pv]

Hi @sokui,
I have tried to make initContainer with --init flag and runtime container without --bootstrap flag.
It looks better and I can see in the UI, that HA is enabled and all 3 nodes are in it.
Here are the logs from init and runtime containers: om_HA_init_success_first_run.zip

However, when I delete the helm release and re-create it (read StatefulSet) without deleting PVCs and re-using them, init container goes into infinite loop, what you can see in the logs here: om_HA_init_infinite_loop.zip
And because of that the runtime container never starts. It is not a big issue for now, because upgrading release within updated Helm chart can work. Only configuration loss or re-creation on Kubernetes can be critical then, because it won't be possible to re-create OM cluster with existing configuration.

Nevertheless, a client is still not able to connect to OM, because traffic redirection from Kubernetes service into the container does not work. The reason remains the same Kubernetes does not redirect traffic if an app is listening on external IP address.

Maybe I need to create a bug ticket about that?

[sokui]

Hi @dantalian-pv ,

What I do for the --init is: first we put all ozone generated configs into PVC, so that next time even we remove the statefulset but as long as the PVC is there, we can start from the last states. For example, we config these properties (for dn, scm and all): hdds.datanode.dir=/data/storage, ozone.scm.datanode.id.dir=/data, and ozone.metadata.dirs=/data/metadata. And the /data/ folder is using PVC volume.

Secondly, in our script, we have some if else judgement to make sure we only run ozone om --init when /data/metadata/om/current/VERSION file exists (similar for scm).

For your connectivity issues, I am not sure what is the root cause. One thing I noticed that you set OZONE-SITE.XML_ozone.om.address: 0.0.0.0:9862, but in our config, we do not have this line. Also can you share the error logs? Does it show the same message as before, or there are different error logs?

[dantalian-pv]

I took some time to find a right combination of --init, --bootstrap and empty set of flags to have a stable deployment for OM and SCM 😅 .
Here is my helm chart (need to rename from .zip to .tgz because github does not allow tgs):
ozone-0.2.1.zip

Here is my current Kubernetes deployment with ConfigMap and Services:
om_HA_kubernetes.zip

With HA deployment OZONE-SITE.XML_ozone.om.address: 0.0.0.0:9862 has no effect, nevertheless I have removed it.

With current deployment I have 2 different problems:

1. When a client is deployed on the same namespace as Apache Ozone, it's able to connect to the OM over Kubernetise Service (name: ozz-ozone-om-svc), but gets the next error:

INFO  RetryInvocationHandler:422 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.OMNotLeaderException): OM:ozz-ozone-om-0 is not the leader. Could not determine the leader node.
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:248)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createLeaderErrorException(OzoneManagerProtocolServerSideTranslatorPB.java:235)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:228)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:175)
        at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
        at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)
, while invoking $Proxy10.submitRequest over nodeId=null,nodeAddress=ozz-ozone-om-svc.test-namespace.svc.cluster.local:9862 after 3 failover attempts. Trying to failover after sleeping for 8000ms. Current retry count: 3.

Even though at the ozz-ozone-om-0 UI shows the next:

OM Roles (HA)	{ HostName: ozz-ozone-om-0.ozz-ozone-om.test-namespace.svc.cluster.local | Node-Id: ozz-ozone-om-0 | Ratis-Port : 9872 | Role: FOLLOWER} { HostName: ozz-ozone-om-1.ozz-ozone-om.test-namespace.svc.cluster.local | Node-Id: ozz-ozone-om-1 | Ratis-Port : 9872 | Role: FOLLOWER} { HostName: ozz-ozone-om-2.ozz-ozone-om.test-namespace.svc.cluster.local | Node-Id: ozz-ozone-om-2 | Ratis-Port : 9872 | Role: LEADER}
Current-Role	FOLLOWER
Group-Id	group-690613BDBFAF

2. When I am trying to port-forward RPC port from any OM Pod, and then use a local client to upload files I have the next error:

$> kubectl -n test-namespace port-forward pod/ozz-ozone-om-0 9862:rpc
Forwarding from 127.0.0.1:9862 -> 9862
Handling connection for 9862
E0731 11:06:20.134283   10843 portforward.go:407] an error occurred forwarding 9862 -> 9862: error forwarding port 9862 to pod 61a897a96023a3fa7592281207c88aab896bf9c29ae9f666e8a72f5af24cf05e, uid : exit status 1: 2023/07/31 09:06:20 socat[1328519] E connect(5, AF=2 127.0.0.1:9862, 16): Connection refused
E0731 11:06:20.134817   10843 portforward.go:233] lost connection to pod

And on the client side:

$> ./hdfs dfs -fs ofs://localhost:9862/ -ls -R /
2023-07-31 11:06:22,146 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost:9862 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy13.submitRequest over nodeId=null,nodeAddress=localhost:9862 after 1 failover attempts. Trying to failover after sleeping for 4000ms. Current retry count: 1.

Port-forwarding is necessary for me, otherwise I have no other possibility to upload files to remote FS.

And at the same time there is no issue to port-forward UI port from the same Pod, or port-forward RPC from the deployment where HA is disabled and only one OM node is deployed.

[dantalian-pv]

Any thoughts, @sokui ☝️ ?

[sokui]

Hi @dantalian-pv ,

My ozone is setup in k8s with HA and with Kerberos enabled. So I am not sure the exact reasons for your case. It seems to me that the leader election of Ratis has some issues. One thing you can test is that if you can go inside the pod, and use ozone cli to upload or download files. Basically you need to do the following:

kubectl exec -ti <your pod> -c <the om container> bash
ozone sh

The second command will list all the sub-commands of command sh, you can do operations with bucket, key, volume, and etc.
Try to go to your leader om first to test this. After that you can go to the follower om to test it again, to see what happened. This will test if your om leader works, and if your om HA works.