Skip to content

Commit

Permalink
[SPARK-44996][K8S] Use lazy val for DefaultVolcanoClient
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to improve `VolcanoFeatureStep` to use lazy creation for `DefaultVolcanoClient`.

### Why are the changes needed?

We can verify this change in the existing test case of `VolcanoFeatureStepSuite`.
Since `VolcanoFeatureStep` creates `DefaultVolcanoClient` always, the unit test suite `VolcanoFeatureStepSuite` behaves like an integration test. In other words, it fails when there is no accessible K8s clusters.
```
$ build/sbt -Pkubernetes -Pvolcano "kubernetes/testOnly *Volcano* -- -z SPARK-36061"
...
[info] VolcanoFeatureStepSuite:
[info] - SPARK-36061: Driver Pod with Volcano PodGroup *** FAILED *** (646 milliseconds)
[info]   org.snakeyaml.engine.v2.exceptions.ScannerException: mapping values are not allowed here
[info]  in reader, line 1, column 94:
[info]      ... well-known/openid-configuration": dial tcp: lookup iam.corp. ...
[info]
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manual tests. Disable network and run the following.
```
$ build/sbt -Pkubernetes -Pvolcano "kubernetes/testOnly *Volcano* -- -z SPARK-36061"
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42710 from dongjoon-hyun/SPARK-44996.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
dongjoon-hyun committed Aug 29, 2023
1 parent df63adf commit 7315a04
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ private[spark] class VolcanoFeatureStep extends KubernetesDriverCustomFeatureCon
"for executor.")
return Seq.empty
}
val client = new DefaultVolcanoClient
lazy val client = new DefaultVolcanoClient
val template = kubernetesConf.getOption(POD_GROUP_TEMPLATE_FILE_KEY)
val pg = template.map(client.podGroups.load(_).item).getOrElse(new PodGroup())
var metadata = pg.getMetadata
Expand Down

0 comments on commit 7315a04

Please sign in to comment.