[SPARK-2208] fix zero shuffle wait time in fast machine

[XuefengWu]

No description provided.

[AmplabJenkins]

Can one of the admins verify this patch?

[rxin]

cc @aarondav

[aarondav]

I am not familiar with this issue or fix. Could you describe in a bit more detail what the intended solution is?

[XuefengWu]

this unit test is try to verify the local metrics, and the shuffleReadMetrics measure time fetch data.
It dependence on number of partitions and read time.
But In my desktop, when the number of partitions is the some as cores, the shuffle read time alway be zero. So I hacked the blockManager, it always sleep 1 millis, to make sure shuffleReadMetrics fetchWaitTime always more than zero.

[XuefengWu]

@aarondav any more suggestion ?

[aarondav]

nit: import ordering should abide by the style guide

[aarondav]

Actually, I realized it might simplify this patch a lot if instead of extending BlockManager, we extend SortShuffleManager which returns a ShuffleBlockManager that injects a wait into getBlockData.

This way you could just do

val myConf = conf.duplicate
  .set("spark.shuffle.manager", clasOf[SlowShuffleManager].getName)
sc = new SparkContext(myConf)

rather than doing all this work creating SparkEnvs and such, which will be annoying any time we add a new parameter to BlockManager or SparkEnv.

Note that I would also refactor this whole suite to construct a single conf once:
val conf = new SparkConf().setMaster("local").setAppName("SparkListenerSuite")

and in each test to just use
sc = new SparkContext(conf)

(rather than in before)

This would allow constructing special SparkContexts without having to have 2 simultaneously like you do with sc2.

[XuefengWu]

@aarondav , thanks point out the style issue, and I think set SlowShuffleManager is a better idea too. thanks.