Securing Ozone with kerberos

[Dalamar32]

When starting up SCM I keep getting this error:

2023-06-07 11:42:20,826 [main] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.io.IOException: DestHost:destPort ddl07oscm03.root.local:9863 , LocalHost:localPort ddl07oscm02.root.local/10.236.152.50:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Kerberos principal name does NOT have the expected hostname part: [email protected], while invoking $Proxy14.send over nodeId=scm3,nodeAddress=ddl07oscm03.root.local/10.236.152.51:9863 after 9 failover attempts. Trying to failover after sleeping for 2000ms.

This error only happens on the other two in the three node cluster, not the 01 node.

According to the documentation on securing ozone it shows the principle name example as "e.g. scm/[email protected]"

Q: Does that mean that the AD account created to use with ozone needs to be named as follows:

scm/_ [email protected]

does it require the host SPN set or is there another issue that I may be unaware of?

Here is my ozone-site.xml file if needed:

ozone-site.txt

[Dalamar32]

Update:

I was not using the proper _HOST wildcard in my ozone-site.xml.

After setting the UPN and SPN to the om/FQDN@Realm and scm/FQDN@Realm scm and om were able to login with their respective UPN. The error I am encoring now is:

2023-07-20 18:30:51,256 [main] WARN ipc.Client: Couldn't setup connection for om/[email protected] to ddl07zone02.root.local/10.236.152.242:9863
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:376)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
        at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:843)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:839)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:839)
        at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1677)
        at org.apache.hadoop.ipc.Client.call(Client.java:1502)
        at org.apache.hadoop.ipc.Client.call(Client.java:1455)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:122)
        at com.sun.proxy.$Proxy31.send(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy31.send(Unknown Source)
        at org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB.submitRequest(ScmBlockLocationProtocolClientSideTranslatorPB.java:121)
        at org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB.getScmInfo(ScmBlockLocationProtocolClientSideTranslatorPB.java:266)
        at org.apache.hadoop.hdds.utils.HAUtils.getScmInfo(HAUtils.java:100)
        at org.apache.hadoop.ozone.om.OzoneManager.omInit(OzoneManager.java:1233)
        at org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.init(OzoneManagerStarter.java:204)
        at org.apache.hadoop.ozone.om.OzoneManagerStarter.initOm(OzoneManagerStarter.java:102)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1972)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
        at picocli.CommandLine.execute(CommandLine.java:2078)
        at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
        at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
        at org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:58)

The key tabs are all correctly setup and work from what I can tell by using kinit to verify that they are still valid.

Anyone have an idea of what this error could be?

[kerneltime]

This still points to a config issue around kerberos. What is the om principal configured for secure API access?

[Dalamar32]

The om principal (UPN) is om/[email protected]

Here is an updated ozone-site.xml file

ozone-site.txt

[Dalamar32]

On the SCM I see this error in the logs:

2023-07-27 14:16:40,058 [Socket Reader #1 for port 9863] WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.236.152.241:38175:null (GSS initiate failed) with true cause: (GSS initiate failed)

[MerelyOneMax1986]

@Dalamar32 run into the same issue in Ozone 1.4.0

According to the example of Ozone HA secure in compose folder of Ozone distribution (as well in HDFS), one may use, for instance, the same principal like scm/scm@REALM for all SCMs in HA cluster.

But in practice, it doesn't work properly.

Is your question is actual for now?

I am open to communication about this topic or any Ozone topic.