-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-25200][YARN] Allow specifying HADOOP_CONF_DIR as spark property #407
Conversation
This reverts commit f8e828c.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One minor comment re style otherwise looks good
val confDirsEnvKeys = Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR") | ||
val confDirProp = sparkConf.getOption("spark.yarn.conf.dir") | ||
|
||
val confDirPaths = (confDirsEnvKeys.map(sys.env.get) :+ confDirProp).flatMap(_.toList) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can just do .flatten
here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
Going to repush to this repo so that tests run |
Thanks for the swift review @robert3005 ! I made the changes |
SPARK-25200 - apache#22289
Replicates the behaviour of setting
HADOOP_CONF_DIR
, but as a spark configspark.yarn.conf.dir