[SPARK-3596][YARN]Support changing the yarn client monitor interval #5292
Conversation
| @@ -125,6 +125,7 @@ private[spark] class YarnClientSchedulerBackend( | |||
| */ | |||
| private def asyncMonitorApplication(): Unit = { | |||
| assert(client != null && appId != null, "Application has not been submitted yet!") | |||
| val interval = conf.getLong("spark.yarn.client.progress.pollinterval", 1000) | |||
There was a problem hiding this comment.
I would change this to something like spark.yarn.client.amPollInterval.
There was a problem hiding this comment.
Thank you, but as it is the client to get the application report from the RM, so maybe "spark.yarn.client.progress.pollinterval" is better.
There was a problem hiding this comment.
My main concern is that we don't really have a notion of "progress" in Spark on YARN. In MapReduce, progress traditionally referred to how far along the app was, but Spark doesn't provide a similar notion. YARN itself does have a notion of app progress, which is not related to what's here.
There was a problem hiding this comment.
Yeah, you are right. In PR #5305 I use the client.monitorApplication, then we can use "spark.yarn.report.interval" to changing the yarn client monitor interval. Thank you.
There was a problem hiding this comment.
OK, is this superseded by #5305 ? it sounds like you have two different changes related to SPARK-3596 and I'm wondering whether they are mutually exclusive, or one or both is preferred, or what.
There was a problem hiding this comment.
Up to your judgment; you're able to open and close PRs.
|
test this please |
|
Jenkins, test this please |
1. YarnClientSchedulerBack.asyncMonitorApplication use Client.monitorApplication so that commonize the monitor logic 2. Support changing the yarn client monitor interval, see #5292 3. More details see discussion on #3143 Author: unknown <[email protected]> Author: Sephiroth-Lin <[email protected]> Closes #5305 from Sephiroth-Lin/SPARK-4346_3596 and squashes the following commits: 47c0014 [unknown] Edit conflicts 52b29fe [unknown] Interrupt thread when we call stop() d4298a1 [unknown] Unused, don't push aaacb42 [Sephiroth-Lin] don't wrap the entire block in the try ee2b2fd [Sephiroth-Lin] update 6483a2a [unknown] Catch exception 6b47ff7 [unknown] Update code 568f46f [unknown] YarnClientSchedulerBack.asyncMonitorApplication should be common with Client.monitorApplication
No description provided.