Skip to content

Commit

Permalink
[SPARK-35136] Remove initial null value of LiveStage.info
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
To prevent potential NullPointerExceptions, this PR changes the `LiveStage` constructor to take `info` as a constructor parameter and adds a nullcheck in  `AppStatusListener.activeStages`.

### Why are the changes needed?
The `AppStatusListener.getOrCreateStage` would create a LiveStage object with the `info` field set to null and right after that set it to a specific StageInfo object. This can lead to a race condition when the `livestages` are read in between those calls. This could then lead to a null pointer exception in, for instance: `AppStatusListener.activeStages`.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Regular CI/CD tests

Closes #32233 from sander-goos/SPARK-35136-livestage.

Authored-by: Sander Goos <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
  • Loading branch information
sander-goos authored and cloud-fan committed Apr 19, 2021
1 parent 718f395 commit a2b5fb3
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -1018,7 +1018,7 @@ private[spark] class AppStatusListener(
*/
def activeStages(): Seq[v1.StageData] = {
liveStages.values.asScala
.filter(_.info.submissionTime.isDefined)
.filter(s => Option(s.info).exists(_.submissionTime.isDefined))
.map(_.toApi())
.toList
.sortBy(_.stageId)
Expand Down Expand Up @@ -1179,7 +1179,7 @@ private[spark] class AppStatusListener(

private def getOrCreateStage(info: StageInfo): LiveStage = {
val stage = liveStages.computeIfAbsent((info.stageId, info.attemptNumber),
(_: (Int, Int)) => new LiveStage())
(_: (Int, Int)) => new LiveStage(info))
stage.info = info
stage
}
Expand Down
3 changes: 1 addition & 2 deletions core/src/main/scala/org/apache/spark/status/LiveEntity.scala
Original file line number Diff line number Diff line change
Expand Up @@ -393,14 +393,13 @@ private class LiveExecutorStageSummary(

}

private class LiveStage extends LiveEntity {
private class LiveStage(var info: StageInfo) extends LiveEntity {

import LiveEntityHelpers._

var jobs = Seq[LiveJob]()
var jobIds = Set[Int]()

var info: StageInfo = null
var status = v1.StageStatus.PENDING

var description: Option[String] = None
Expand Down

0 comments on commit a2b5fb3

Please sign in to comment.