Secure HDFS Support requires most recent PRs

[ifilonenko]

Upon investigation of Secure HDFS support, I have found that the recent PRs that have supported moving Delegation Token Renewal logic into Spark Core are instrumental towards providing a clean implementation.
The PR of focus is this one: SPARK-20434: Move Hadoop delegation token code from yarn to core by mgummelt · Pull Request #17723 · apache/spark

Which is an initial step for this PR: SPARK-16742: Mesos Kerberos Support by mgummelt · Pull Request #18519 · apache/spark. Because we will be re-using alot of this logic, what is the strategy in re-using most recent commits instead of using ugly reflections to access private methods in private packages (if that is even possible).

This issue is in reference to this PR: #373

[dimberman]

It sounds like you're creating a circular dependency.

The yarn package depends on the spark-core package, any attempts to pull that code back in to spark-core will expose you to bugs in the future and be a nightmare of reflection and code injection.

There are a few solutions I can think of based on this issue:

1. Pull whatever dependencies you need back into spark-core (though judging by our IRL conversation, that might not go over so well)
2. Create a new package that has dependencies on spark-core and yarn
3. Move the hadoop delegation code into its own package and modify the YARN package to depend on the hadoop-delegation-token package.

Personally, 2 and 3 seem like the best stop-gap solutions, which can easily transition into a more ideal set-up at a future date.