Skip to content

Commit

Permalink
[SPARK-29580][TESTS] Add kerberos debug messages for Kafka secure tests
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
`org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite` failed lately. After had a look at the logs it just shows the following fact without any details:
```
Caused by: sbt.ForkMain$ForkError: sun.security.krb5.KrbException: Server not found in Kerberos database (7) - Server not found in Kerberos database
```
Since the issue is intermittent and not able to reproduce it we should add more debug information and wait for reproduction with the extended logs.

### Why are the changes needed?
Failing test doesn't give enough debug information.

### Does this PR introduce any user-facing change?
No.

### How was this patch tested?
I've started the test manually and checked that such additional debug messages show up:
```
>>> KrbApReq: APOptions are 00000000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
Looking for keys for: kafka/localhostEXAMPLE.COM
Added key: 17version: 0
Added key: 23version: 0
Added key: 16version: 0
Found unsupported keytype (3) for kafka/localhostEXAMPLE.COM
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
Using builtin default etypes for permitted_enctypes
default etypes for permitted_enctypes: 17 16 23.
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
MemoryCache: add 1571936500/174770/16C565221B70AAB2BEFE31A83D13A2F4/client/localhostEXAMPLE.COM to client/localhostEXAMPLE.COM|kafka/localhostEXAMPLE.COM
MemoryCache: Existing AuthList:
#3: 1571936493/200803/8CD70D280B0862C5DA1FF901ECAD39FE/client/localhostEXAMPLE.COM
#2: 1571936499/985009/BAD33290D079DD4E3579A8686EC326B7/client/localhostEXAMPLE.COM
#1: 1571936499/995208/B76B9D78A9BE283AC78340157107FD40/client/localhostEXAMPLE.COM
```

Closes apache#26252 from gaborgsomogyi/SPARK-29580.

Authored-by: Gabor Somogyi <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
gaborgsomogyi authored and dongjoon-hyun committed Oct 25, 2019
1 parent 2baf7a1 commit 2549391
Showing 1 changed file with 21 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ class KafkaTestUtils(
secure: Boolean = false) extends Logging {

private val JAVA_AUTH_CONFIG = "java.security.auth.login.config"
private val IBM_KRB_DEBUG_CONFIG = "com.ibm.security.krb5.Krb5Debug"
private val SUN_KRB_DEBUG_CONFIG = "sun.security.krb5.debug"

private val localCanonicalHostName = InetAddress.getLoopbackAddress().getCanonicalHostName()
logInfo(s"Local host name is $localCanonicalHostName")
Expand Down Expand Up @@ -133,6 +135,7 @@ class KafkaTestUtils(
private def setUpMiniKdc(): Unit = {
val kdcDir = Utils.createTempDir()
val kdcConf = MiniKdc.createConf()
kdcConf.setProperty(MiniKdc.DEBUG, "true")
kdc = new MiniKdc(kdcConf, kdcDir)
kdc.start()
kdcReady = true
Expand Down Expand Up @@ -238,6 +241,7 @@ class KafkaTestUtils(
}

if (secure) {
setupKrbDebug()
setUpMiniKdc()
val jaasConfigFile = createKeytabsAndJaasConfigFile()
System.setProperty(JAVA_AUTH_CONFIG, jaasConfigFile)
Expand All @@ -252,6 +256,14 @@ class KafkaTestUtils(
}
}

private def setupKrbDebug(): Unit = {
if (System.getProperty("java.vendor").contains("IBM")) {
System.setProperty(IBM_KRB_DEBUG_CONFIG, "all")
} else {
System.setProperty(SUN_KRB_DEBUG_CONFIG, "true")
}
}

/** Teardown the whole servers, including Kafka broker and Zookeeper */
def teardown(): Unit = {
if (leakDetector != null) {
Expand Down Expand Up @@ -303,6 +315,15 @@ class KafkaTestUtils(
kdc.stop()
}
UserGroupInformation.reset()
teardownKrbDebug()
}

private def teardownKrbDebug(): Unit = {
if (System.getProperty("java.vendor").contains("IBM")) {
System.clearProperty(IBM_KRB_DEBUG_CONFIG)
} else {
System.clearProperty(SUN_KRB_DEBUG_CONFIG)
}
}

/** Create a Kafka topic and wait until it is propagated to the whole cluster */
Expand Down

0 comments on commit 2549391

Please sign in to comment.