[Train] Add Intel HPU Support and Training Examples to Ray Train

[kira-lin]

Why are these changes needed?

To leverage the potential of Intel's Habana Processing Units (HPUs), we extend Ray Train's capabilities by adding support for Intel HPU hardware. This update also includes training scripts for BERT and ResNet models, which are widely used in NLP and computer vision tasks, respectively.

Related issue number

Replace #42866

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[woshiyyya]

@kira-lin Can you address my comments in your previous PR #42866?

[kira-lin]

@woshiyyya I've addressed all of them, please review.

[woshiyyya]

@kira-lin Thanks for the hard work. The PR looks much better now.

I left some comments on the example and please take a look. It should be good to go after we polished the wordings in the examples.

[kira-lin]

@woshiyyya addressed

[woshiyyya]

LGTM. Thanks for the contribution!

@kira-lin We will put the examples into Community Example section. In the future, if there is a user issue related to these examples, I'll hand it over to you.

[matthewdeng]

Should this import logic be moved to within _setup_torch_process_group and only called when backend == "hccl"? Seems like this is needed to be called on the workers.

[kira-lin]

updated

[matthewdeng]

@kira-lin can you push the change to the PR?

[kira-lin]

sorry, forgot 🥲 pushed

[matthewdeng]

Looks like there is a merge conflict for this file, could you take a look at resolving it?

[kira-lin]

resolved

[matthewdeng]

Looks like the doc build is failing. Would you be able to take a look? I can also take a look later if needed.

[kira-lin]

sorry, not sure how to fix it

[liuxsh9]

It's great to see that Ray Train has successfully extended its support beyond GPUs to other acceleration hardware. Interestingly, we are also preparing to contribute support for Huawei NPU. The adaptation process for torch_npu is very similar to torch_hpu.
However, we are concerned that following the HPU approach to add NPU support might make the code more complex and introduce scattered device type and library checks.
Therefore, in terms of design, we would like to separate the hardware-specific components, similar to the abstraction of accelerators, to make it easier for Ray Train (even RLlib) to support third-party devices.
We have already made some designs that ensure it won't affect the usage of GPUs and HPUs, and we provide references for integrating third-party devices with APIs like prepare_model and prepare_data_loader. We would love to hear the community's opinions on this matter.
Should we initiate a PR to showcase the implementation (including some code migration for HPU), or would it be better to initiate an issue first to discuss this matter? @kira-lin @woshiyyya @matthewdeng

[woshiyyya]

Hi @liuxsh9 , you are welcome to add support for NPU accelerator in Ray Train: ) If you already have the designs, feel free to post a PR and we can discuss the implementation details.

[anyscalesam]

@angelinalg can we pick this into latest; I think it might have just missed the branch cut date so right now the "latest" URL renders to a 404.

Switching to master works fine though.

https://docs.ray.io/en/latest/train/examples/hpu/resnet.html
https://docs.ray.io/en/master/train/examples/hpu/resnet.html

[anyscalesam]

@liuxsh9 we should also have the discussion at the Core layer first as the foundational interface to custom chipsets start with Core with Ray Libs building on top of that.

Let's connect online Xiaoshuang.