Support ROCm

[HinaHyugaHime]

🚀 Feature

Support ROCm on AI generation

Motivation

would like to be able to use xformers on my linux rocm install of stable diffusion

Pitch

Alternatives

Additional context

[danthe3rd]

Hi,
We don't have plans to support ROCm at the moment, and I also assume it would be a non-trivial amount of work. We also don't have a way to develop on AMD GPUs, nor do we have the ability to run tests for these devices on our CI.
We can accept contribution if someone wants to have a look at it.

[tedliosu]

Hi, We don't have plans to support ROCm at the moment, and I also assume it would be a non-trivial amount of work. We also don't have a way to develop on AMD GPUs, nor do we have the ability to run tests for these devices on our CI. We can accept contribution if someone wants to have a look at it.

May I know what will be the minimum amount of changes that it'll take for the dev team to be able to develop and run tests on AMD GPUs? Is Nvidia also directly helping the dev team in acquiring up-to-date hardware to develop and run tests on for the Nvidia side of things? 👀

[danthe3rd]

Hi @tedliosu
We do have some support from NVIDIA, mostly on the software side (CUTLASS etc...). To be honest with you, there are multiple blockers at the moment to supporting AMD GPUs:
(1) there is non-trivial amount of work to write kernels that would work on these GPUs. We can't just convert to AMD GPUs the CUTLASS kernels. Triton might be a solution tho, but most likely in the longer term
(2) it adds some complexity in terms of maintenance (we also need ROCm builds as well as CUDA builds). More kernels means more failure points. We would also need to setup tests in the CI with various AMD GPUs, and provide support to our users. Also, when developing a new feature for NVIDIA devices, we would need to do additional work to make it work on AMD
(3) we don't have any AMD hardware in the xFormers team at the moment, while we have plenty of V100/A100 (and more recently H100) to play with
(4) and most importantly, the teams we support internally all use NVIDIA GPUs, so this line of work would not benefit any of the research teams we work with inside Meta, while requiring a big amount of work on our side

[tedliosu]

Hi @tedliosu We do have some support from NVIDIA, mostly on the software side (CUTLASS etc...). To be honest with you, there are multiple blockers at the moment to supporting AMD GPUs: (1) there is non-trivial amount of work to write kernels that would work on these GPUs. We can't just convert to AMD GPUs the CUTLASS kernels. Triton might be a solution tho, but most likely in the longer term (2) it adds some complexity in terms of maintenance (we also need ROCm builds as well as CUDA builds). More kernels means more failure points. We would also need to setup tests in the CI with various AMD GPUs, and provide support to our users. Also, when developing a new feature for NVIDIA devices, we would need to do additional work to make it work on AMD (3) we don't have any AMD hardware in the xFormers team at the moment, while we have plenty of V100/A100 (and more recently H100) to play with (4) and most importantly, the teams we support internally all use NVIDIA GPUs, so this line of work would not benefit any of the research teams we work with inside Meta, while requiring a big amount of work on our side

Thank you for the clarification. So what you're saying basically boils down to supporting an additional GPU vendor not only being at least twice the amount of work on the devs' end, but also that there's a lack of "market share" of AMD GPUs at least amongst those companies that the dev team work with?

Also, I remember reading amongst the issues here that pull requests for supporting xformers for AMD GPUs are welcome? Especially since the devs currently have no way of testing any code on AMD GPUs, will any pull requests that add support for AMD GPUs also need to include additional AMD-specific tests from the person submitting them to ensure their correctness?

[danthe3rd]

Ideally we would want to have a discussion with the person first, as this might be quite some work, and we need to evaluate the best way moving forward in that direction. But in principle we welcome contributions :)

[HinaHyugaHime]

Ideally we would want to have a discussion with the person first, as this might be quite some work, and we need to evaluate the best way moving forward in that direction. But in principle we welcome contributions :)

elaborate who you mean "the person" and amd has the best marketshare on gaming wise, but nvidia obviously on AI and lately overall because of AI and overall compatability leaning towards nvidia for AI, but ROCm is making strides to bringing ROCm to everything currently on linux all the AMD GPU's are supported, however on Windows only the latest AMD GPU's are, it would more than likely be just to do with Triton I agree, but ROCm has open source repos which can be found here for the runtimes and so on https://github.com/RadeonOpenCompute it is literally a wrapper for CUDA

[HinaHyugaHime]

I have made multiple guides that I have merged into 2 to installing ROCm on linux for different platforms

[danthe3rd]

elaborate who you mean "the person"

Someone who has both the knowledge and time to add support for ROCm + maintain it going forward

My understanding is that ROCm is able to compile CUDA code into something that can run on AMD GPUs. However, this is not possible for the CUDA code we write, because we rely on third-party libraries like CUTLASS who use inline-PTX which is not compilable with ROCm. But if you find a way to get our kernels to run with ROCm we can discuss

[Iron-Bound]

Seem's AMD has forked his to add support https://github.com/ROCm/xformers/

[danthe3rd]

We're working with AMD to add minimal support for AMD GPUs to xFormers:
#978
This will be largely best-effort, and we don't plan yet to invest more to make it easy to use (eg pre-built wheels etc...)
cc @qianfengz

[HinaHyugaHime]

We're working with AMD to add minimal support for AMD GPUs to xFormers: #978 This will be largely best-effort, and we don't plan yet to invest more to make it easy to use (eg pre-built wheels etc...) cc @qianfengz

so it wont be something I can use by using pip install? or how would that work

[danthe3rd]

Most likely you will need to build from source (which is also possible via pip, but takes more time), and it would only support inference at first

[HinaHyugaHime]

can you update me when the pull goes through?

[danthe3rd]

Hum I believe you should be able to subscribe to the PR #978

[HinaHyugaHime]

is this going to be a ongoing project for a while? (to finish the first testable)