Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[IntelliJ][ReportedByUser] Azure Plugin for IntelliJ Error - Submit to spark23 cluster fails with ADLS gen 1 #2316

Closed
jingyanjingyan opened this issue Oct 30, 2018 · 3 comments
Closed

[IntelliJ][ReportedByUser] Azure Plugin for IntelliJ Error - Submit to spark23 cluster fails with ADLS gen 1 #2316

jingyanjingyan opened this issue Oct 30, 2018 · 3 comments
Assignees
@jingyanjingyan
Labels
Module: HDInsight Tool: IntelliJ Type: Feature Request
Milestone
v3.16.0

Comments

@jingyanjingyan
Copy link

IntelliJ build version: 2018.2.5 IC-182.4892.20
OS: Windows 10
JDK: JetBrains s.r.o 1.8.0_152-release
Plugin version: 3.14.0
Error message:

com.microsoft.azure.hdinsight.spark.common.SparkJobException: The Spark job failed to start due to 18/10/30 14:11:34 INFO ShutdownHookManager: Shutdown hook called
18/10/30 14:11:34 INFO ShutdownHookManager: Deleting directory /tmp/spark-795714ed-ed32-4f45-9bd3-4ef944f3007f
18/10/30 14:11:34 INFO MetricsSystemImpl: Stopping azure-file-system metrics system...
18/10/30 14:11:34 INFO MetricsSinkAdapter: azurefs2 thread interrupted.
18/10/30 14:11:34 INFO MetricsSystemImpl: azure-file-system metrics system stopped.
18/10/30 14:11:34 INFO MetricsSystemImpl: azure-file-system metrics system shutdown complete.

stderr:

YARN Diagnostics:
java.lang.Exception: No YARN application is found with tag livy-batch-99-scgm1zlj in 120 seconds. Please check your cluster status, it is may be very busy.
org.apache.livy.utils.SparkYarnApp.org$apache$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:239) org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:236) scala.Option.getOrElse(Option.scala:120) org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:236) org.apache.livy.Utils$$anon$1.run(Utils.scala:94)
at com.microsoft.azure.hdinsight.spark.common.SparkBatchJob.lambda$awaitStarted$56(SparkBatchJob.java:1221)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at rx.internal.operators.OnSubscribeFilter$FilterSubscriber.onNext(On

========================================================
Build: PR 2438
Repro Steps:

  1. Set adls gen 1 settings
    image

  2. Submit to spark23hdinsight/spark23test
    Result:
    image

com.microsoft.azure.hdinsight.spark.common.SparkJobException: The Spark job failed to start due to 18/10/30 14:11:34 INFO ShutdownHookManager: Shutdown hook called
18/10/30 14:11:34 INFO ShutdownHookManager: Deleting directory /tmp/spark-795714ed-ed32-4f45-9bd3-4ef944f3007f
18/10/30 14:11:34 INFO MetricsSystemImpl: Stopping azure-file-system metrics system...
18/10/30 14:11:34 INFO MetricsSinkAdapter: azurefs2 thread interrupted.
18/10/30 14:11:34 INFO MetricsSystemImpl: azure-file-system metrics system stopped.
18/10/30 14:11:34 INFO MetricsSystemImpl: azure-file-system metrics system shutdown complete.

stderr:

YARN Diagnostics:
java.lang.Exception: No YARN application is found with tag livy-batch-99-scgm1zlj in 120 seconds. Please check your cluster status, it is may be very busy.
org.apache.livy.utils.SparkYarnApp.org$apache$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:239) org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:236) scala.Option.getOrElse(Option.scala:120) org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:236) org.apache.livy.Utils$$anon$1.run(Utils.scala:94)
at com.microsoft.azure.hdinsight.spark.common.SparkBatchJob.lambda$awaitStarted$56(SparkBatchJob.java:1221)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at rx.internal.operators.OnSubscribeFilter$FilterSubscriber.onNext(OnSubscribeFilter.java:76)
at rx.internal.operators.OperatorTakeUntilPredicate$ParentSubscriber.onNext(OperatorTakeUntilPredicate.java:61)
at rx.internal.operators.OnSubscribeRedo$2$1.onNext(OnSubscribeRedo.java:244)
at rx.internal.operators.OnSubscribeRedo$2$1.onNext(OnSubscribeRedo.java:244)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleDelayedProducer.emit(SingleDelayedProducer.java:102)
at rx.internal.producers.SingleDelayedProducer.setValue(SingleDelayedProducer.java:85)
at rx.internal.operators.OnSubscribeFromCallable.call(OnSubscribeFromCallable.java:48)
at rx.internal.operators.OnSubscribeFromCallable.call(OnSubscribeFromCallable.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeRedo$2.call(OnSubscribeRedo.java:273)
at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)
at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)
at rx.internal.operators.OnSubscribeRedo$5.request(OnSubscribeRedo.java:361)
at rx.internal.producers.ProducerArbiter.setProducer(ProducerArbiter.java:126)
at rx.internal.operators.OnSubscribeRedo$2$1.setProducer(OnSubscribeRedo.java:267)
at rx.internal.operators.OnSubscribeRedo.call(OnSubscribeRedo.java:353)
at rx.internal.operators.OnSubscribeRedo.call(OnSubscribeRedo.java:47)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeRedo$2.call(OnSubscribeRedo.java:273)
at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)
at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)
at rx.internal.operators.OnSubscribeRedo$5.request(OnSubscribeRedo.java:361)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeRedo.call(OnSubscribeRedo.java:353)
at rx.internal.operators.OnSubscribeRedo.call(OnSubscribeRedo.java:47)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeFilter.call(OnSubscribeFilter.java:45)
at rx.internal.operators.OnSubscribeFilter.call(OnSubscribeFilter.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:395)
at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:355)
at rx.internal.operators.OperatorMerge$InnerSubscriber.onNext(OperatorMerge.java:846)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onNext(OnSubscribeDoOnEach.java:101)
at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:395)
at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:355)
at rx.internal.operators.OperatorMerge$InnerSubscriber.onNext(OperatorMerge.java:846)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OperatorSingle$ParentSubscriber.onCompleted(OperatorSingle.java:110)
at rx.internal.operators.DeferredScalarSubscriber.complete(DeferredScalarSubscriber.java:102)
at rx.internal.operators.OnSubscribeTakeLastOne$TakeLastOneSubscriber.onCompleted(OnSubscribeTakeLastOne.java:57)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onCompleted(OnSubscribeDoOnEach.java:70)
at com.microsoft.azure.hdinsight.spark.common.SparkBatchJob.lambda$getSubmissionLog$33(SparkBatchJob.java:898)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153)
at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32)
at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.operators.OperatorSubscribeOn$SubscribeOnSubscriber.onNext(OperatorSubscribeOn.java:74)
at rx.internal.producers.SingleDelayedProducer.emit(SingleDelayedProducer.java:102)
at rx.internal.producers.SingleDelayedProducer.setValue(SingleDelayedProducer.java:85)
at rx.internal.operators.OnSubscribeFromCallable.call(OnSubscribeFromCallable.java:48)
at rx.internal.operators.OnSubscribeFromCallable.call(OnSubscribeFromCallable.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorSubscribeOn$SubscribeOnSubscriber.call(OperatorSubscribeOn.java:100)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
at rx.internal.schedulers.ExecutorScheduler$ExecutorSchedulerWorker.run(ExecutorScheduler.java:107)
at com.microsoft.intellij.rxjava.IdeaSchedulers$1.run(IdeaSchedulers.java:50)
at com.intellij.openapi.progress.impl.CoreProgressManager$TaskRunnable.run(CoreProgressManager.java:736)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$1(CoreProgressManager.java:157)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:580)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:525)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:85)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:144)
at com.intellij.openapi.progress.impl.CoreProgressManager$4.run(CoreProgressManager.java:395)
at com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:314)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
@t-rufang
Copy link
Contributor

If an HDInsight cluster has no ADLS storage account binded, the submission with ADLS Gen 1 storage type will definitely fail. In future we will disable the unavailable storage type when cluster is selected.

@t-rufang t-rufang added this to the v3.15.0 milestone Oct 31, 2018
@wezhang wezhang modified the milestones: v3.15.0, v3.16.0 Nov 21, 2018
@wezhang wezhang assigned lcadzy and unassigned t-rufang Dec 10, 2018
@lcadzy lcadzy mentioned this issue Dec 12, 2018
Merged
@lcadzy
Copy link
Contributor

lcadzy commented Dec 12, 2018

adls storage type should not be available for this cluster

@lcadzy lcadzy assigned jingyanjingyan and unassigned lcadzy Dec 12, 2018
@jingyanjingyan
Copy link
Author

Close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jingyanjingyan jingyanjingyan

Labels
Module: HDInsight Tool: IntelliJ Type: Feature Request
Projects
None yet
Milestone
v3.16.0
Development

No branches or pull requests
4 participants
@wezhang @t-rufang @jingyanjingyan @lcadzy