[SPARK-34820][K8S][R] add apt-update before gnupg install #31923

Yikun · 2021-03-22T07:51:39Z

What changes were proposed in this pull request?

We added the gnupg installation in #30130 , we should do apt update before gnupg isntallation, otherwise we will get a fetch error when package is updated.

See more in:
[1] http://apache-spark-developers-list.1001551.n3.nabble.com/K8s-Integration-test-is-unable-to-run-because-of-the-unavailable-libs-td30986.html

Why are the changes needed?

add a apt-update cmd before gnupg installation to avoid invaild package cache list.

Does this PR introduce any user-facing change?

No

How was this patch tested?

K8s Integration test passed

Yikun · 2021-03-22T08:01:17Z

cc @Ngone51

Ngone51 · 2021-03-22T08:04:02Z

@Yikun thanks for the quick fix. cc @dongjoon-hyun @holdenk

Ngone51 · 2021-03-22T08:07:25Z

ok to test

SparkQA · 2021-03-22T08:23:50Z

Test build #136335 has finished for PR 31923 at commit 7418fdb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-22T09:03:27Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40920/

SparkQA · 2021-03-22T09:41:14Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40920/

Yikun · 2021-03-22T12:16:39Z

Step 6/12 : RUN   apt-get update &&   apt install -y gnupg &&   echo "deb http://cloud.r-project.org/bin/linux/debian buster-cran35/" >> /etc/apt/sources.list &&   (apt-key adv --keyserver keys.gnupg.net --recv-key 'E19F5F87128899B192B1A2C2AD5F960A256A04AF' || apt-key adv --keyserver keys.openpgp.org --recv-key 'E19F5F87128899B192B1A2C2AD5F960A256A04AF') &&   apt-get update &&   apt install -y -t buster-cran35 r-base r-base-dev &&   rm -rf /var/cache/apt/*
 ---> Running in 890d910fdc12
... ...

 ---> 6fc29805545e
Step 7/12 : COPY R ${SPARK_HOME}/R
 ---> 98781dd121aa
Step 8/12 : ENV R_HOME /usr/lib/R

From [1] (Step 6/12 succeed), we can see the failed fetch problem of SPARK-34820 had been fixed.

[1] https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40920/consoleFull

Yikun · 2021-03-22T12:25:55Z

- Run SparkR on simple dataframe.R example *** FAILED ***
  io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://192.168.39.219:8443/api/v1/namespaces/5b12a03f6e2c40308f8c12bf4d9ea3e3/pods/spark-test-app-f99d40879bc94e569c8ae32b79a88970/log?pretty=false. Message: container "spark-kubernetes-driver" in pod "spark-test-app-f99d40879bc94e569c8ae32b79a88970" is waiting to start: trying and failing to pull image. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=container "spark-kubernetes-driver" in pod "spark-test-app-f99d40879bc94e569c8ae32b79a88970" is waiting to start: trying and failing to pull image, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=BadRequest, status=Failure, additionalProperties={}).
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:570)
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:509)
  at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.doGetLog(PodOperationsImpl.java:189)
  at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.getLog(PodOperationsImpl.java:198)
  at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.getLog(PodOperationsImpl.java:85)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.$anonfun$logForFailedTest$3(KubernetesSuite.scala:89)
  at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
  at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
  at org.apache.spark.SparkFunSuite.logInfo(SparkFunSuite.scala:61)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.$anonfun$logForFailedTest$2(KubernetesSuite.scala:86)
  ...

and there are still some other errors triggerd, looks like unrelated.

Ngone51 · 2021-03-22T13:18:53Z

retest this please

Ngone51 · 2021-03-22T13:19:13Z

Let's retry test see if it's flaky

SparkQA · 2021-03-22T13:35:46Z

Test build #136345 has finished for PR 31923 at commit 7418fdb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-22T14:14:55Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40929/

SparkQA · 2021-03-22T14:47:33Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40929/

Yikun · 2021-03-22T15:00:27Z

The SparkPullRequestBuilder-K8s (Kubernetes integration test) back to green again, I think the PR is ready to merge.

[1] https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/

shaneknapp · 2021-03-22T15:40:04Z

thanks for doing this!

dongjoon-hyun

+1, LGTM.
Merged to master/3.1.
cc @attilapiros

### What changes were proposed in this pull request? We added the gnupg installation in #30130 , we should do apt update before gnupg isntallation, otherwise we will get a fetch error when package is updated. See more in: [1] http://apache-spark-developers-list.1001551.n3.nabble.com/K8s-Integration-test-is-unable-to-run-because-of-the-unavailable-libs-td30986.html ### Why are the changes needed? add a apt-update cmd before gnupg installation to avoid invaild package cache list. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? K8s Integration test passed Closes #31923 from Yikun/SPARK-34820. Authored-by: Yikun Jiang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 31da907) Signed-off-by: Dongjoon Hyun <[email protected]>

Ngone51 · 2021-03-25T02:00:32Z

Hi all, I'd like to get your continued attention on the K8s integration test issue. After this PR fixed the gnupg installation issue, another issue shows up, which is almost a constant failure:

- Run SparkR on simple dataframe.R example *** FAILED ***
  io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://192.168.39.219:8443/api/v1/namespaces/b80cb1250aba4d92a99fc87b609b2328/pods/spark-test-app-411d038edc8b4e9b8e3761052cd44bd8/log?pretty=false. Message: container "spark-kubernetes-driver" in pod "spark-test-app-411d038edc8b4e9b8e3761052cd44bd8" is waiting to start: trying and failing to pull image. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=container "spark-kubernetes-driver" in pod "spark-test-app-411d038edc8b4e9b8e3761052cd44bd8" is waiting to start: trying and failing to pull image, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=BadRequest, status=Failure, additionalProperties={}).
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:570)
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:509)
  at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.doGetLog(PodOperationsImpl.java:189)
  at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.getLog(PodOperationsImpl.java:198)
  at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.getLog(PodOperationsImpl.java:85)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.$anonfun$logForFailedTest$3(KubernetesSuite.scala:89)
  at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
  at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
  at org.apache.spark.SparkFunSuite.logInfo(SparkFunSuite.scala:61)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.$anonfun$logForFailedTest$2(KubernetesSuite.scala:86)

And here is PR that has tried K8s integration test multiple times but all failed.
Could someone please help take a look? Thanks!

attilapiros · 2021-03-25T12:32:37Z

@Ngone51 I checked the PR you mentioned.

My findings based on the last failure.

Here the first error is:

- Launcher client dependencies
- SPARK-33615: Launcher client archives *** FAILED ***
  io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://192.168.39.147:8443/api/v1/namespaces/09e9c94160d543c1a338f364722d49a6/pods/spark-test-app-c80c78f512574b36b2608f7d92c24503/log?pretty=false. Message: container "spark-kubernetes-driver" in pod "spark-test-app-c80c78f512574b36b2608f7d92c24503" is waiting to start: trying and failing to pull image. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=container "spark-kubernetes-driver" in pod "spark-test-app-c80c78f512574b36b2608f7d92c24503" is waiting to start: trying and failing to pull image, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=BadRequest, status=Failure, additionalProperties={}).
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:570)
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:509)
  at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.doGetLog(PodOperationsImpl.java:189)
  at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.getLog(PodOperationsImpl.java:198)
  at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.getLog(PodOperationsImpl.java:85)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.$anonfun$logForFailedTest$3(KubernetesSuite.scala:89)
  at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
  at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
  at org.apache.spark.SparkFunSuite.logInfo(SparkFunSuite.scala:61)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.$anonfun$logForFailedTest$2(KubernetesSuite.scala:86)
  ...

Checking the last successful and first failed ones we can see just a very few differences in the code: the failed one uses a --archives and the successful uses --files.

I do not think this difference could lead to an error such "is waiting to start: trying and failing to pull image."

attilapiros · 2021-03-25T13:05:08Z

It would be wonderful to see kubectl describe pod <pod> output.
Actually I think I can do this in one of my PR...

attilapiros · 2021-03-25T14:48:59Z

Of course locally all the tests passed:

All tests passed.
...
[INFO] Spark Project Kubernetes Integration Tests ......... SUCCESS [25:12 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

attilapiros · 2021-03-25T16:03:17Z

I hope this will help to troubleshoot this and similar errors:
#31962

Ngone51 · 2021-03-29T12:43:13Z

Thanks for the effort @attilapiros

shaneknapp · 2021-03-29T23:53:52Z

fwiw the k8s and minikube installs on the workers haven't changed... i went through earlier today and ensured that we didn't have any dangling minikube processes on the workers, and killed any leftover build procs as well. i'm not sure why this is happening... also, we'll be upgrading both k8s and mk in the next week or so.

…

On Mon, Mar 29, 2021 at 5:43 AM wuyi ***@***.***> wrote: Thanks for the effort @attilapiros <https://github.com/attilapiros> — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#31923 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMIHLCLOTD75VQB646XATTTGBYXHANCNFSM4ZSS2MKQ> .

### What changes were proposed in this pull request? We added the gnupg installation in apache#30130 , we should do apt update before gnupg isntallation, otherwise we will get a fetch error when package is updated. See more in: [1] http://apache-spark-developers-list.1001551.n3.nabble.com/K8s-Integration-test-is-unable-to-run-because-of-the-unavailable-libs-td30986.html ### Why are the changes needed? add a apt-update cmd before gnupg installation to avoid invaild package cache list. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? K8s Integration test passed Closes apache#31923 from Yikun/SPARK-34820. Authored-by: Yikun Jiang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 31da907) Signed-off-by: Dongjoon Hyun <[email protected]>

Move the apt-update before apt install

7418fdb

github-actions bot added KUBERNETES R labels Mar 22, 2021

srowen approved these changes Mar 22, 2021

View reviewed changes

Ngone51 approved these changes Mar 22, 2021

View reviewed changes

shaneknapp self-requested a review March 22, 2021 15:38

shaneknapp approved these changes Mar 22, 2021

View reviewed changes

dongjoon-hyun approved these changes Mar 22, 2021

View reviewed changes

dongjoon-hyun closed this in 31da907 Mar 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-34820][K8S][R] add apt-update before gnupg install #31923

[SPARK-34820][K8S][R] add apt-update before gnupg install #31923

Yikun commented Mar 22, 2021 •

edited

Loading

Yikun commented Mar 22, 2021

Ngone51 commented Mar 22, 2021

Ngone51 commented Mar 22, 2021

SparkQA commented Mar 22, 2021

SparkQA commented Mar 22, 2021

SparkQA commented Mar 22, 2021

Yikun commented Mar 22, 2021

Yikun commented Mar 22, 2021

Ngone51 commented Mar 22, 2021

Ngone51 commented Mar 22, 2021

SparkQA commented Mar 22, 2021

SparkQA commented Mar 22, 2021

SparkQA commented Mar 22, 2021

Yikun commented Mar 22, 2021

shaneknapp commented Mar 22, 2021

dongjoon-hyun left a comment

Ngone51 commented Mar 25, 2021

attilapiros commented Mar 25, 2021

attilapiros commented Mar 25, 2021

attilapiros commented Mar 25, 2021

attilapiros commented Mar 25, 2021

Ngone51 commented Mar 29, 2021

shaneknapp commented Mar 29, 2021 via email

[SPARK-34820][K8S][R] add apt-update before gnupg install #31923

[SPARK-34820][K8S][R] add apt-update before gnupg install #31923

Conversation

Yikun commented Mar 22, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Yikun commented Mar 22, 2021

Ngone51 commented Mar 22, 2021

Ngone51 commented Mar 22, 2021

SparkQA commented Mar 22, 2021

SparkQA commented Mar 22, 2021

SparkQA commented Mar 22, 2021

Yikun commented Mar 22, 2021

Yikun commented Mar 22, 2021

Ngone51 commented Mar 22, 2021

Ngone51 commented Mar 22, 2021

SparkQA commented Mar 22, 2021

SparkQA commented Mar 22, 2021

SparkQA commented Mar 22, 2021

Yikun commented Mar 22, 2021

shaneknapp commented Mar 22, 2021

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Ngone51 commented Mar 25, 2021

attilapiros commented Mar 25, 2021

attilapiros commented Mar 25, 2021

attilapiros commented Mar 25, 2021

attilapiros commented Mar 25, 2021

Ngone51 commented Mar 29, 2021

shaneknapp commented Mar 29, 2021 via email

Yikun commented Mar 22, 2021 •

edited

Loading