[SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests #31829

attilapiros · 2021-03-14T06:38:01Z

What changes were proposed in this pull request?

This PR upgrades Kubernetes and Minikube version for integration tests and removes/updates the old code for this new version.

Details of this changes:

As discussed in the mailing list: updating Minikube version from v0.34.1 to v1.7.3 and kubernetes version from v1.15.12 to v1.17.3.
making Minikube version checked and fail with an explanation when the test is started with on a version < v1.7.3.
removing minikube status checking code related to old Minikube versions
in the Minikube backend using fabric8's Config.autoConfigure() method to configure the kubernetes client to use the minikube k8s context (like it was in one of the Minikube's example)
Introducing persistentVolume test tag: this would be a temporary change to skip PVC tests in the Kubernetes integration test, as currently the PCV tests are blocking the move to Docker as Minikube's driver (for details please check https://issues.apache.org/jira/browse/SPARK-34738).

Why are the changes needed?

With the current suggestion one can run into several problems without noticing the Minikube/kubernetes version is the problem.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

It was tested on Mac with this script which installs each Minikube versions from v1.7.2 (including this version to test the negative case of the version check) and runs the integration tests.

It was started with:

./check_minikube_versions.zsh > test_log 2>&1

And there was only one build failure the rest was successful:

$ grep "BUILD SUCCESS" test_log | wc -l
      26
$ grep "BUILD FAILURE" test_log | wc -l
       1

It was for Minikube v1.7.2 and the log is:

KubernetesSuite:
*** RUN ABORTED ***
  java.lang.AssertionError: assertion failed: Unsupported Minikube version is detected: minikube version: v1.7.2.For integration testing Minikube version 1.7.3 or greater is expected.
  at scala.Predef$.assert(Predef.scala:223)
  at org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.getKubernetesClient(Minikube.scala:52)
  at org.apache.spark.deploy.k8s.integrationtest.backend.minikube.MinikubeTestBackend$.initialize(MinikubeTestBackend.scala:33)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:163)
  at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
  at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
  at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.org$scalatest$BeforeAndAfter$$super$run(KubernetesSuite.scala:43)
  at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
  at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
  ...

Moreover I made a test with having multiple k8s cluster contexts, too.

SparkQA · 2021-03-14T07:21:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40621/

SparkQA · 2021-03-14T07:26:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40621/

rvesse

LGTM, one comment about needing a doc update around the requirement for the context to be called minikube

...s/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala

fbalicchia · 2021-04-06T17:37:20Z

Hi, I'm experiencing in the same issue and and to resolve the issue I've applied the same changes. Could you please merge it ?
Thanks

attilapiros · 2021-04-06T19:53:07Z

@fbalicchia Could you please share with us the error you run into (the one on which this PR helped on)?
By the way I cannot merge my own PR without an expert's approval (expert of the area where the PR belongs to). But your input will help the case.

fbalicchia · 2021-04-06T20:23:12Z

Using minikube v1.18.1 default driver is Docker and at the moment of writing masterUrl not necessary bind on port 8443 https://github.com/apache/spark/blob/master/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala#L69.
Instead using default config or using factory with autoConfigure("minikube") library help us

attilapiros · 2021-04-08T10:00:20Z

resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh

@@ -176,4 +180,4 @@ properties+=(
  -Dlog4j.logger.org.apache.spark=DEBUG
 )

-$TEST_ROOT_DIR/build/mvn integration-test -f $TEST_ROOT_DIR/pom.xml -pl resource-managers/kubernetes/integration-tests -am -Pscala-$SCALA_VERSION -P$HADOOP_PROFILE -Pkubernetes -Pkubernetes-integration-tests ${properties[@]}
+$TEST_ROOT_DIR/build/mvn install -f $TEST_ROOT_DIR/pom.xml -pl resource-managers/kubernetes/integration-tests $BUILD_DEPENDENCIES_MVN_FLAG -Pscala-$SCALA_VERSION -P$HADOOP_PROFILE -Pkubernetes -Pkubernetes-integration-tests ${properties[@]}


The install phase is after integration-test, see default lifecycle at the maven doc:

Phase Description

... ...

package take the compiled code and package it in its distributable format, such as a JAR.

pre-integration-test perform actions required before integration tests are executed. This may involve things such as setting up the required environment.

integration-test process and deploy the package if necessary into an environment where integration tests can be run.

post-integration-test perform actions required after integration tests have been executed. This may including cleaning up the environment.

verify run any checks to verify the package is valid and meets quality criteria.

install install the package into the local repository, for use as a dependency in other projects locally.

... ...

With install the local repo will be updated with the fresh jars and in case of --skip-building-dependencies the build will reuse those.

Well, I know that we need install sometimes, but for this one, I'm a little negative for converting integration-test to install because it pollutes the local repo while testing and reviewing someone-else's PR. It's really painful for committers to clean up the local repo everytime to avoid any side-effects for the other PRs. Is this really inevitable?

I would say this is the orherway around, assuming in both PRs we stick to install.
So when we call install in the first PR we could be sure the fresh jars will be used (even when the artifact we build referencing something outside of the maven reactor) and when we move to the next PR because of install we have again the fresh jars.

SparkQA · 2021-04-08T10:28:12Z

Test build #137071 has finished for PR 31829 at commit de32835.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-08T10:57:11Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41649/

SparkQA · 2021-04-08T11:02:51Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41649/

SparkQA · 2021-04-17T05:02:41Z

Test build #137499 has finished for PR 31829 at commit 1f594aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-17T05:27:47Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42073/

github-actions · 2021-04-17T07:57:25Z

Test build #758035203 for PR 31829 at commit 855273a.

attilapiros · 2021-04-17T07:57:34Z

rebased on master

SparkQA · 2021-04-17T08:11:19Z

Test build #137502 has finished for PR 31829 at commit 855273a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-17T08:40:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42076/

SparkQA · 2021-04-17T08:40:24Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42076/

SparkQA · 2021-04-19T05:22:56Z

Test build #137566 has finished for PR 31829 at commit 855273a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-19T10:15:13Z

Test build #137602 has finished for PR 31829 at commit 855273a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

@attilapiros . Is there an ETA for the blocker you mentioned? If there is no ETA for that, it would be great if we can proceed a subset first partially. For those parts which are out of our control, we had better reduce the bottleneck or dependency as much as possible.

BTW, Spark 3.2 code freeze is July 1st 2021. So, we still have enough dev time though.

attilapiros · 2021-04-30T10:52:33Z

@dongjoon-hyun actually I am pretty sure this helps (even with the skipping of the PVTests which when @shaneknapp sets the Minikube driver to Docker will fix the Kubernetes integration tests).

So could you please review this as it is?

dongjoon-hyun · 2021-05-09T05:07:43Z

Oh, I missed your last comment here. Very sorry, @attilapiros .

resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh

...s/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala

...ation-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala

dongjoon-hyun · 2021-05-09T05:28:30Z

I left a few comments.

[SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests #31829 (comment)
[SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests #31829 (comment)
[SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests #31829 (comment)
[SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests #31829 (comment)

dongjoon-hyun

+1, LGTM Thank you, @attilapiros and all.

Sorry for back and forth. On second thought, for the above all my comments, we can handle them later when we need them. Let's merge this and move forward. Thanks!

attilapiros · 2021-05-10T16:49:15Z

Thanks @dongjoon-hyun.

Tested locally:

[INFO] --- scalatest-maven-plugin:2.0.0:test (integration-test) @ spark-kubernetes-integration-tests_2.12 ---
Discovery starting.
Discovery completed in 446 milliseconds.
Run starting. Expected test count is: 25
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- Launcher client dependencies
- SPARK-33615: Launcher client archives
- SPARK-33748: Launcher python client respecting PYSPARK_PYTHON
- SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python
- Launcher python client dependencies using a zip file
- Test basic decommissioning
- Test basic decommissioning with shuffle cleanup
- Test decommissioning with dynamic allocation & shuffle cleanups
- Test decommissioning timeouts
Run completed in 16 minutes, 51 seconds.
Total number of tests run: 25

github-actions bot added DOCS KUBERNETES labels Mar 14, 2021

attilapiros mentioned this pull request Mar 14, 2021

[SPARK-32617][K8S][TESTS] Configure kubernetes client based on kubeconfig settings in kubernetes integration tests #30751

Closed

rvesse approved these changes Mar 15, 2021

View reviewed changes

...s/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala Show resolved Hide resolved

attilapiros force-pushed the SPARK-34736 branch from f643bd9 to de32835 Compare April 8, 2021 09:53

attilapiros commented Apr 8, 2021

View reviewed changes

attilapiros added 2 commits April 17, 2021 09:57

Intial upload

d46d0ea

add persistentVolume tag for PVTests

855273a

attilapiros force-pushed the SPARK-34736 branch from 1f594aa to 855273a Compare April 17, 2021 07:57

attilapiros mentioned this pull request Apr 30, 2021

[MINOR][DOC] Bump minikube minimum version to 1.7.3 in README.md #32403

Closed

dongjoon-hyun reviewed Apr 30, 2021

View reviewed changes

attilapiros changed the title ~~[WIP][SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests~~ [SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests Apr 30, 2021

dongjoon-hyun reviewed May 9, 2021

View reviewed changes

resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh Show resolved Hide resolved

dongjoon-hyun reviewed May 9, 2021

View reviewed changes

...s/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala Show resolved Hide resolved

dongjoon-hyun reviewed May 9, 2021

View reviewed changes

...ation-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala Show resolved Hide resolved

dongjoon-hyun approved these changes May 9, 2021

View reviewed changes

attilapiros closed this in 8b94eff May 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests #31829

[SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests #31829

attilapiros commented Mar 14, 2021 •

edited

Loading

SparkQA commented Mar 14, 2021

SparkQA commented Mar 14, 2021

rvesse left a comment

fbalicchia commented Apr 6, 2021 •

edited

Loading

attilapiros commented Apr 6, 2021

fbalicchia commented Apr 6, 2021

attilapiros Apr 8, 2021

dongjoon-hyun May 9, 2021

attilapiros May 9, 2021

SparkQA commented Apr 8, 2021

SparkQA commented Apr 8, 2021

SparkQA commented Apr 8, 2021

SparkQA commented Apr 17, 2021

SparkQA commented Apr 17, 2021

github-actions bot commented Apr 17, 2021

attilapiros commented Apr 17, 2021

SparkQA commented Apr 17, 2021

SparkQA commented Apr 17, 2021

SparkQA commented Apr 17, 2021

SparkQA commented Apr 19, 2021

SparkQA commented Apr 19, 2021

dongjoon-hyun left a comment •

edited

Loading

attilapiros commented Apr 30, 2021

dongjoon-hyun commented May 9, 2021

dongjoon-hyun commented May 9, 2021

dongjoon-hyun left a comment •

edited

Loading

attilapiros commented May 10, 2021

Phase	Description
...	...
package	take the compiled code and package it in its distributable format, such as a JAR.
pre-integration-test	perform actions required before integration tests are executed. This may involve things such as setting up the required environment.
integration-test	process and deploy the package if necessary into an environment where integration tests can be run.
post-integration-test	perform actions required after integration tests have been executed. This may including cleaning up the environment.
verify	run any checks to verify the package is valid and meets quality criteria.
install	install the package into the local repository, for use as a dependency in other projects locally.
...	...

[SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests #31829

[SPARK-34736][K8S][TESTS] Kubernetes and Minikube version upgrade for integration tests #31829

Conversation

attilapiros commented Mar 14, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Mar 14, 2021

SparkQA commented Mar 14, 2021

rvesse left a comment

Choose a reason for hiding this comment

fbalicchia commented Apr 6, 2021 • edited Loading

attilapiros commented Apr 6, 2021

fbalicchia commented Apr 6, 2021

attilapiros Apr 8, 2021

Choose a reason for hiding this comment

dongjoon-hyun May 9, 2021

Choose a reason for hiding this comment

attilapiros May 9, 2021

Choose a reason for hiding this comment

SparkQA commented Apr 8, 2021

SparkQA commented Apr 8, 2021

SparkQA commented Apr 8, 2021

SparkQA commented Apr 17, 2021

SparkQA commented Apr 17, 2021

github-actions bot commented Apr 17, 2021

attilapiros commented Apr 17, 2021

SparkQA commented Apr 17, 2021

SparkQA commented Apr 17, 2021

SparkQA commented Apr 17, 2021

SparkQA commented Apr 19, 2021

SparkQA commented Apr 19, 2021

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

attilapiros commented Apr 30, 2021

dongjoon-hyun commented May 9, 2021

dongjoon-hyun commented May 9, 2021

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

attilapiros commented May 10, 2021

attilapiros commented Mar 14, 2021 •

edited

Loading

fbalicchia commented Apr 6, 2021 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading