From 7315a046e22e1e19f2f3cf723d1a135e2baef089 Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Mon, 28 Aug 2023 22:13:21 -0700 Subject: [PATCH] [SPARK-44996][K8S] Use `lazy val` for `DefaultVolcanoClient` ### What changes were proposed in this pull request? This PR aims to improve `VolcanoFeatureStep` to use lazy creation for `DefaultVolcanoClient`. ### Why are the changes needed? We can verify this change in the existing test case of `VolcanoFeatureStepSuite`. Since `VolcanoFeatureStep` creates `DefaultVolcanoClient` always, the unit test suite `VolcanoFeatureStepSuite` behaves like an integration test. In other words, it fails when there is no accessible K8s clusters. ``` $ build/sbt -Pkubernetes -Pvolcano "kubernetes/testOnly *Volcano* -- -z SPARK-36061" ... [info] VolcanoFeatureStepSuite: [info] - SPARK-36061: Driver Pod with Volcano PodGroup *** FAILED *** (646 milliseconds) [info] org.snakeyaml.engine.v2.exceptions.ScannerException: mapping values are not allowed here [info] in reader, line 1, column 94: [info] ... well-known/openid-configuration": dial tcp: lookup iam.corp. ... [info] ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual tests. Disable network and run the following. ``` $ build/sbt -Pkubernetes -Pvolcano "kubernetes/testOnly *Volcano* -- -z SPARK-36061" ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #42710 from dongjoon-hyun/SPARK-44996. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- .../apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala index 1a06f7ee0bf6f..314550713ef16 100644 --- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala +++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala @@ -46,7 +46,7 @@ private[spark] class VolcanoFeatureStep extends KubernetesDriverCustomFeatureCon "for executor.") return Seq.empty } - val client = new DefaultVolcanoClient + lazy val client = new DefaultVolcanoClient val template = kubernetesConf.getOption(POD_GROUP_TEMPLATE_FILE_KEY) val pg = template.map(client.podGroups.load(_).item).getOrElse(new PodGroup()) var metadata = pg.getMetadata