[SPARK-35013][CORE] Don't allow to set spark.driver.cores=0

[shahidki31]

What changes were proposed in this pull request?

Currently spark is not allowing to set spark.driver.memory, spark.executor.cores, spark.executor.memory to 0, but allowing driver cores to 0. This PR checks for driver core size as well. Thanks @oleg Lypkan for finding this.

Why are the changes needed?

To make the configuration check consistent.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manual testing

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42906/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42906/

[SparkQA]

Test build #138383 has finished for PR 32504 at commit b615db6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Apache Spark depends on your GitHub Repo's GitHub Action jobs.

• https://spark.apache.org/developer-tools.html

Could you update your master branch and your PR branch once more and enable all GitHub Actions jobs in your repo like the following?

• https://github.com/apache/spark/actions

[shahidki31]

Sure @dongjoon-hyun . Let me go through the doc. Thanks

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42918/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42920/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42920/

[SparkQA]

Test build #138396 has finished for PR 32504 at commit 89a2796.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #138398 has finished for PR 32504 at commit a63fcc3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shahidki31]

@dongjoon-hyun I have updated the branch with master. Kindly review

[HyukjinKwon]

@shahidki31 can you retrigger the test please? https://github.com/shahidki31/spark/actions/runs/832876763

[HyukjinKwon]

cc @Ngone51 @mridulm @tgravescs FYI

[shahidki31]

@shahidki31 can you retrigger the test please? https://github.com/shahidki31/spark/actions/runs/832876763

Sure @HyukjinKwon Let me check

[shahidki31]

Hi @HyukjinKwon All the tests are passing. Seems Linters, Licence job is failing. But that is unrelated to the PR. Is it a known failure?

[HyukjinKwon]

That's fixed in the upstream. Can you sync with the latest master branch of Apache Spark repo by rebasing/merging?

[shahidki31]

Thanks. Let me retrgger that.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42965/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42965/

[SparkQA]

Test build #138444 has finished for PR 32504 at commit 5f480e7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42972/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42972/

[SparkQA]

Test build #138451 has finished for PR 32504 at commit a26b180.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tgravescs]

changes look fine, assume it was manually tested

[shahidki31]

changes look fine, assume it was manually tested

@tgravescs Yes, I verified manually

[shahidki31]

Retest this please

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42986/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42986/

[dongjoon-hyun]

+1, LGTM.

[dongjoon-hyun]

Thank you, @shahidki31 and all.
Merged to master.

[SparkQA]

Test build #138465 has finished for PR 32504 at commit a26b180.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shahidki31]

Thanks @dongjoon-hyun and all for the reviews