[SPARK-36462][K8S] Add the ability to selectively disable watching or polling #36433

holdenk · 2022-05-02T19:31:55Z

What changes were proposed in this pull request?

Add the ability to selectively disable watching or polling

Updated version of #34264

Why are the changes needed?

Watching or polling for pod status on Kubernetes can place additional load on etcd, with a large number of executors and large number of jobs this can have negative impacts and executors register themselves with the driver under normal operations anyways.

Does this PR introduce any user-facing change?

Two new config flags.

How was this patch tested?

New unit tests + manually tested a forked version of this on an internal cluster with both watching and polling disabled.

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

martin-g · 2022-05-03T11:40:43Z

.../src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSource.scala


  private var watchConnection: Closeable = _
+  private val enablePolling = conf.get(KUBERNETES_EXECUTOR_ENABLE_API_WATCHER)


enableWatching ?

.../src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSource.scala

...test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSourceSuite.scala

holdenk · 2022-05-03T21:49:25Z

Thanks @martin-g for the review :)

Yikun

CI failed is legit due to tiny style error. otherwise LGTM.

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

dongjoon-hyun

Hi, @holdenk . Could you address the above comments and make CIs happy?

…n Kubernetes for environments where etcd may be under a high load or otherwise not support polling/watching.

…ark/deploy/k8s/Config.scala Co-authored-by: Martin Grigorov <[email protected]>

…ark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSourceSuite.scala Co-authored-by: Martin Grigorov <[email protected]>

…ark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSource.scala Co-authored-by: Martin Grigorov <[email protected]>

holdenk · 2022-08-11T22:36:28Z

CC @dongjoon-hyun I believe done

dongjoon-hyun · 2022-08-12T04:19:09Z

Thank you, @holdenk .

dongjoon-hyun · 2022-08-12T04:20:44Z

.../src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSource.scala

@@ -38,19 +40,23 @@ import org.apache.spark.util.Utils
 @DeveloperApi
 class ExecutorPodsWatchSnapshotSource(
    snapshotsStore: ExecutorPodsSnapshotsStore,
-    kubernetesClient: KubernetesClient) extends Logging {
+    kubernetesClient: KubernetesClient,
+    conf: SparkConf) extends Logging {


Oh, this is a developer API as Stable level. Could you make a separate constructor for this new feature?
I believe we should keep the existing 2-parameter constructer for backward compatibility.

dongjoon-hyun

@holdenk . I left one comment about backward compatibility. For the other stuffs, looks good.

https://github.com/apache/spark/pull/36433/files#r944102080

…unning sc to grab conf if no conf specified.

dongjoon-hyun

+1, LGTM. Thank you, @holdenk and all.
Sorry for being late. Merged to master for Apache Spark 3.4.0.

… polling ### What changes were proposed in this pull request? Add the ability to selectively disable watching or polling Updated version of apache#34264 ### Why are the changes needed? Watching or polling for pod status on Kubernetes can place additional load on etcd, with a large number of executors and large number of jobs this can have negative impacts and executors register themselves with the driver under normal operations anyways. ### Does this PR introduce _any_ user-facing change? Two new config flags. ### How was this patch tested? New unit tests + manually tested a forked version of this on an internal cluster with both watching and polling disabled. Closes apache#36433 from holdenk/SPARK-36462-allow-spark-on-kube-to-operate-without-watchers. Lead-authored-by: Holden Karau <[email protected]> Co-authored-by: Holden Karau <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 5bffb98)

* ODP-2189 Upgrade snakeyaml version to 2.0 * [SPARK-35579][SQL] Bump janino to 3.1.7 ### What changes were proposed in this pull request? upgrade janino to 3.1.7 from 3.0.16 ### Why are the changes needed? - The proposed version contains bug fix in janino by maropu. - janino-compiler/janino#148 - contains `getBytecodes` method which can be used to simplify the way to get bytecodes from ClassBodyEvaluator in CodeGenerator#updateAndGetCompilationStats method. (by LuciferYang) - apache#32536 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing UTs Closes apache#37202 from singhpk234/upgrade/bump-janino. Authored-by: Prashant Singh <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit 29ed337) * [SPARK-40633][BUILD] Upgrade janino to 3.1.9 ### What changes were proposed in this pull request? This pr aims upgrade janino from 3.1.7 to 3.1.9 ### Why are the changes needed? This version bring some improvement and bug fix, and janino 3.1.9 will no longer test Java 12, 15, 16 because these STS versions have been EOL: - janino-compiler/janino@v3.1.7...v3.1.9 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - Manual test this pr with Scala 2.13, all test passed Closes apache#38075 from LuciferYang/SPARK-40633. Lead-authored-by: yangjie01 <[email protected]> Co-authored-by: YangJie <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit 49e102b) * ODP-2167 Upgrade janino version from 3.1.9 to 3.1.10 * ODP-2190 Upgrade guava version to 32.1.3-jre * ODP-2193 Upgrade jettison version to 1.5.4 * ODP-2194 Upgrade wildfly-openssl version to 1.1.3 * ODP-2198 Upgrade gson version to 2.11.0 * ODP-2199 Upgrade kryo-shaded version to 4.0.3 * ODP-2200 Upgrade datanucleus-core and datanucleus-rdbms versions to 5.2.3 * ODP-2203 Upgrade Snappy and common-compress to 1.1.10.4 and 1.26.0 respectively * ODP-2198 Excluded gson from tink library * ODP-2205 Upgrade jdom2 to 2.0.6.1 * ODP-2198 Excluded gson from hive-exec * ODP-2175|SPARK-47018 Upgrade libthrift version and hive version * [SPARK-39688][K8S] `getReusablePVCs` should handle accounts with no PVC permission ### What changes were proposed in this pull request? This PR aims to handle `KubernetesClientException` in `getReusablePVCs` method to handle gracefully the cases where accounts has no PVC permission including `listing`. ### Why are the changes needed? To prevent a regression in Apache Spark 3.4. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs with the newly added test case. Closes apache#37095 from dongjoon-hyun/SPARK-39688. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 79f133b) * [SPARK-40458][K8S] Bump Kubernetes Client Version to 6.1.1 ### What changes were proposed in this pull request? Bump kubernetes-client version from 5.12.3 to 6.1.1 and clean up all the deprecations. ### Why are the changes needed? To keep up with kubernetes-client [changes](fabric8io/kubernetes-client@v5.12.3...v6.1.1). As this is an upgrade where the main version changed I have cleaned up all the deprecations. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? #### Unit tests #### Manual tests for submit and application management Started an application in a non-default namespace (`bla`): ``` ➜ spark git:(SPARK-40458) ✗ ./bin/spark-submit \ --master k8s://http://127.0.0.1:8001 \ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ --conf spark.executor.instances=5 \ --conf spark.kubernetes.namespace=bla \ --conf spark.kubernetes.container.image=docker.io/kubespark/spark:3.4.0-SNAPSHOT_064A99CC-57AF-46D5-B743-5B12692C260D \ local:///opt/spark/examples/jars/spark-examples_2.12-3.4.0-SNAPSHOT.jar 200000 ``` Check that we cannot find it in the default namespace even with glob without the namespace definition: ``` ➜ spark git:(SPARK-40458) ✗ minikube kubectl -- config set-context --current --namespace=default Context "minikube" modified. ➜ spark git:(SPARK-40458) ✗ ./bin/spark-submit --status "spark-pi-*" --master k8s://http://127.0.0.1:8001 Submitting a request for the status of submission spark-pi-* in k8s://http://127.0.0.1:8001. No applications found. ``` Then check we can find it by specifying the namespace: ``` ➜ spark git:(SPARK-40458) ✗ ./bin/spark-submit --status "bla:spark-pi-*" --master k8s://http://127.0.0.1:8001 Submitting a request for the status of submission bla:spark-pi-* in k8s://http://127.0.0.1:8001. Application status (driver): pod name: spark-pi-4c4e70837c86ae1a-driver namespace: bla labels: spark-app-name -> spark-pi, spark-app-selector -> spark-c95a9a0888214c01a286eb7ba23980a0, spark-role -> driver, spark-version -> 3.4.0-SNAPSHOT pod uid: 0be8952e-3e00-47a3-9082-9cb45278ed6d creation time: 2022-09-27T01:19:06Z service account name: default volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-wxnqw node name: minikube start time: 2022-09-27T01:19:06Z phase: Running container status: container name: spark-kubernetes-driver container image: kubespark/spark:3.4.0-SNAPSHOT_064A99CC-57AF-46D5-B743-5B12692C260D container state: running container started at: 2022-09-27T01:19:07Z ``` Changing the namespace to `bla` with `kubectl`: ``` ➜ spark git:(SPARK-40458) ✗ minikube kubectl -- config set-context --current --namespace=bla Context "minikube" modified. ``` Checking we can find it without specifying the namespace (and glob): ``` ➜ spark git:(SPARK-40458) ✗ ./bin/spark-submit --status "spark-pi-*" --master k8s://http://127.0.0.1:8001 Submitting a request for the status of submission spark-pi-* in k8s://http://127.0.0.1:8001. Application status (driver): pod name: spark-pi-4c4e70837c86ae1a-driver namespace: bla labels: spark-app-name -> spark-pi, spark-app-selector -> spark-c95a9a0888214c01a286eb7ba23980a0, spark-role -> driver, spark-version -> 3.4.0-SNAPSHOT pod uid: 0be8952e-3e00-47a3-9082-9cb45278ed6d creation time: 2022-09-27T01:19:06Z service account name: default volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-wxnqw node name: minikube start time: 2022-09-27T01:19:06Z phase: Running container status: container name: spark-kubernetes-driver container image: kubespark/spark:3.4.0-SNAPSHOT_064A99CC-57AF-46D5-B743-5B12692C260D container state: running container started at: 2022-09-27T01:19:07Z ``` Killing the app: ``` ➜ spark git:(SPARK-40458) ✗ ./bin/spark-submit --kill "spark-pi-*" --master k8s://http://127.0.0.1:8001 Submitting a request to kill submission spark-pi-* in k8s://http://127.0.0.1:8001. Grace period in secs: not set. Deleting driver pod: spark-pi-4c4e70837c86ae1a-driver. ``` Closes apache#37990 from attilapiros/SPARK-40458. Authored-by: attilapiros <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit fa88651) * [SPARK-36462][K8S] Add the ability to selectively disable watching or polling ### What changes were proposed in this pull request? Add the ability to selectively disable watching or polling Updated version of apache#34264 ### Why are the changes needed? Watching or polling for pod status on Kubernetes can place additional load on etcd, with a large number of executors and large number of jobs this can have negative impacts and executors register themselves with the driver under normal operations anyways. ### Does this PR introduce _any_ user-facing change? Two new config flags. ### How was this patch tested? New unit tests + manually tested a forked version of this on an internal cluster with both watching and polling disabled. Closes apache#36433 from holdenk/SPARK-36462-allow-spark-on-kube-to-operate-without-watchers. Lead-authored-by: Holden Karau <[email protected]> Co-authored-by: Holden Karau <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 5bffb98) * ODP-2201|SPARK-48867 Upgrade okhttp to 4.12.0, okio to 3.9.0 and esdk-obs-java to 3.24.3 * [SPARK-41958][CORE][3.3] Disallow arbitrary custom classpath with proxy user in cluster mode Backporting fix for SPARK-41958 to 3.3 branch from apache#39474 Below description from original PR. -------------------------- ### What changes were proposed in this pull request? This PR proposes to disallow arbitrary custom classpath with proxy user in cluster mode by default. ### Why are the changes needed? To avoid arbitrary classpath in spark cluster. ### Does this PR introduce _any_ user-facing change? Yes. User should reenable this feature by `spark.submit.proxyUser.allowCustomClasspathInClusterMode`. ### How was this patch tested? Manually tested. Closes apache#39474 from Ngone51/dev. Lead-authored-by: Peter Toth <peter.tothgmail.com> Co-authored-by: Yi Wu <yi.wudatabricks.com> Signed-off-by: Hyukjin Kwon <gurwls223apache.org> (cherry picked from commit 909da96) ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Closes apache#41428 from degant/spark-41958-3.3. Lead-authored-by: Degant Puri <[email protected]> Co-authored-by: Peter Toth <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> * ODP-2049 Changing Spark3 version from 3.3.3.3.2.3.2-2 to 3.3.3.3.2.3.2-201 * ODP-2049 Changing libthrift version to 0.16 in deps files * ODP-2049 Changing derby version to 10.14.3.0 --------- Signed-off-by: Dongjoon Hyun <[email protected]> Co-authored-by: Prashant Singh <[email protected]> Co-authored-by: yangjie01 <[email protected]> Co-authored-by: Dongjoon Hyun <[email protected]> Co-authored-by: attilapiros <[email protected]> Co-authored-by: Holden Karau <[email protected]> Co-authored-by: Degant Puri <[email protected]> Co-authored-by: Peter Toth <[email protected]>

github-actions bot added the KUBERNETES label May 2, 2022

martin-g reviewed May 3, 2022

View reviewed changes

Yikun reviewed May 7, 2022

View reviewed changes

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated Show resolved Hide resolved

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated Show resolved Hide resolved

holdenk force-pushed the SPARK-36462-allow-spark-on-kube-to-operate-without-watchers branch from 3699f9b to 9ae6946 Compare June 28, 2022 21:42

dcoliversun reviewed Jun 29, 2022

View reviewed changes

dongjoon-hyun reviewed Aug 10, 2022

View reviewed changes

holdenk and others added 10 commits August 10, 2022 18:14

Add the ability to selectively disable watching or polling for pods o…

2ed6c6b

…n Kubernetes for environments where etcd may be under a high load or otherwise not support polling/watching.

CR-feedback

efa0e5a

Update resource-managers/kubernetes/core/src/main/scala/org/apache/sp…

eac0698

…ark/deploy/k8s/Config.scala Co-authored-by: Martin Grigorov <[email protected]>

Update resource-managers/kubernetes/core/src/main/scala/org/apache/sp…

a030c0b

…ark/deploy/k8s/Config.scala Co-authored-by: Martin Grigorov <[email protected]>

Update resource-managers/kubernetes/core/src/test/scala/org/apache/sp…

346f40f

…ark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSourceSuite.scala Co-authored-by: Martin Grigorov <[email protected]>

Update resource-managers/kubernetes/core/src/main/scala/org/apache/sp…

6fef69d

…ark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSource.scala Co-authored-by: Martin Grigorov <[email protected]>

CR feedback it's enableWatching not polling update var name to match.

c8523c1

Update version since we missed the boat on 3.3.0

9137c4d

Fix style

f0db5a7

Move internal up above booleanConf

adf4e9a

holdenk force-pushed the SPARK-36462-allow-spark-on-kube-to-operate-without-watchers branch from 9ae6946 to adf4e9a Compare August 11, 2022 01:15

dongjoon-hyun reviewed Aug 12, 2022

View reviewed changes

CR Feedback: add back legacy two parameter constructor, use default r…

9f5d460

…unning sc to grab conf if no conf specified.

dongjoon-hyun approved these changes Aug 18, 2022

View reviewed changes

dongjoon-hyun closed this in 5bffb98 Aug 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-36462][K8S] Add the ability to selectively disable watching or polling #36433

[SPARK-36462][K8S] Add the ability to selectively disable watching or polling #36433

holdenk commented May 2, 2022

martin-g May 3, 2022

holdenk commented May 3, 2022

Yikun left a comment

dongjoon-hyun left a comment

holdenk commented Aug 11, 2022

dongjoon-hyun commented Aug 12, 2022

dongjoon-hyun Aug 12, 2022

holdenk Aug 12, 2022

dongjoon-hyun left a comment

dongjoon-hyun left a comment


		private var watchConnection: Closeable = _
		private val enablePolling = conf.get(KUBERNETES_EXECUTOR_ENABLE_API_WATCHER)

[SPARK-36462][K8S] Add the ability to selectively disable watching or polling #36433

[SPARK-36462][K8S] Add the ability to selectively disable watching or polling #36433

Conversation

holdenk commented May 2, 2022

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

martin-g May 3, 2022

Choose a reason for hiding this comment

holdenk commented May 3, 2022

Yikun left a comment

Choose a reason for hiding this comment

dongjoon-hyun left a comment

Choose a reason for hiding this comment

holdenk commented Aug 11, 2022

dongjoon-hyun commented Aug 12, 2022

dongjoon-hyun Aug 12, 2022

Choose a reason for hiding this comment

holdenk Aug 12, 2022

Choose a reason for hiding this comment

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun left a comment

Choose a reason for hiding this comment