fix: ensure correct version of torch is always installed based on BUI…

[cryptk]

Description

Previously torch was being installed in the requirements.txt file, this would pull in the CUDA12 version of torch, which isn't appropriate for other BUILD_TYPE configurations. This PR moves all of the pytorch related packages to BUILD_TYPE specific files so that we always install the correct pytorch package to match the BUILD_TYPE.

Hopefully this will help with issues such as #2737 and #1592.

Notes for Reviewers

Given that pytorch needs "special handling" based on the build_type, this would be a good candidate to pull the install of pytorch out of the requirements files and into the libbackend.sh library to better handle the torch, torchaudio and torchvision installations. Many python backends have requirements-TYPE.txt files purely for handling the complexities of the torch installation, so it is very easy to get it wrong.

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	1303285
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/66b0e81873551f0008001d67
😎 Deploy Preview	https://deploy-preview-2890--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dave-gray101]

@cryptk I think this is a good idea. However, I think that dependabot may not be picking up our "accelerator specific" files, and torch is obviously pretty important to track (even if in most cases, we're probably stuck on whatever version the project requires)

Based on https://stackoverflow.com/questions/75899186/how-to-configure-dependabot-to-check-multiple-files it looks like we might be better off renaming these files, simply to invert the naming to something like cublas11-requirements.txt

Do you mind double checking me here and potentially doing the rename as a part of this PR since it's already basically touching everything? If you think that's a risky enough change to make its own PR, that's obviously fine too.

[cryptk]

@dave-gray101 once I have some time I'll take a look at your recommendation, I've been pretty swamped lately

[gymnae]

This would be great, because at the moment I cannot use this project on anything but CUDA powered hardware. CPU only AMD 8700GE or CPU & iGPU won't work with the current docker images, even after rebuilding.
Since AMD is also attempting to mainline the driver stack for their (Xilinx) AI silicon for Kernel 6.12, a build type specific proecss would be helpful

[bunder2015]

Hi @gymnae if you are using docker, try using sha-e676809-hipblas-ffmpeg as the image name... I can't find which gfx#### code the 8700ge is at the moment. If I have some spare time this week I might try applying this patch to my local instance, but I've had issues with other changes in the past so no guarantees until the patch can get merged to the repo. Cheers

[gymnae]

Hi @gymnae if you are using docker, try using sha-e676809-hipblas-ffmpeg as the image name... I can't find which gfx#### code the 8700ge is at the moment. If I have some spare time this week I might try applying this patch to my local instance, but I've had issues with other changes in the past so no guarantees until the patch can get merged to the repo. Cheers

Thank you, I will try that. The 8700GE has a 870M iGPU, which is gfx1103 and currently not natively supported by ROCm, but an override works, at least in so far that ROCm is happy

EDIT: Does not improve the situation for me, but I managed to get the pure CPU version running by passing the entire iGPU to the VM running localai, which tells me that I had issues with the host before

[mudler]

Seems to break on mamba:

 #51 108.9 Initializing libbackend for mamba
#51 108.9 virtualenv created
#51 108.9 virtualenv activated
#51 108.9 activated virtualenv has been ensured
#51 108.9 starting requirements install for /build/backend/python/mamba/requirements-install.txt
#51 108.9 finished requirements install for /build/backend/python/mamba/requirements-install.txt
#51 108.9 starting requirements install for /build/backend/python/mamba/requirements.txt
#51 108.9 make: Leaving directory '/build/backend/python/mamba'
#51 108.9 Using Python 3.10.12 interpreter at: /usr/bin/python3
#51 108.9 Creating virtualenv at: venv
#51 108.9 Resolved 3 packages in 3ms
#51 108.9 Installed 3 packages in 13ms
#51 108.9  + packaging==24.1
#51 108.9  + setuptools==72.1.0
#51 108.9  + wheel==0.43.0
#51 108.9 error: Failed to download and build `causal-conv1d==1.4.0`
#51 108.9   Caused by: Failed to build: `causal-conv1d==1.4.0`
#51 108.9   Caused by: Build backend failed to determine metadata through `prepare_metadata_for_build_wheel` with exit status: 1
#51 108.9 --- stdout:
#51 108.9 
#51 108.9 --- stderr:
#51 108.9 Traceback (most recent call last):
#51 108.9   File "<string>", line 14, in <module>
#51 108.9   File "/build/backend/python/mamba/venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 368, in prepare_metadata_for_build_wheel
#51 108.9     self.run_setup()
#51 108.9   File "/build/backend/python/mamba/venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 497, in run_setup
#51 108.9     super().run_setup(setup_script=setup_script)
#51 108.9   File "/build/backend/python/mamba/venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 313, in run_setup
#51 108.9     exec(code, locals())
#51 108.9   File "<string>", line 20, in <module>
#51 108.9 ModuleNotFoundError: No module named 'torch'

[mudler]

I think I've got most of the fixes out, let's see if CI is now liking it

[mudler]

Should be available in master images as CI builds and publish the images - @bunder2015 @gymnae if you can test that now everything works as expected that'd be appreciated!

[bunder2015]

Hi @mudler I waited for quay to output the ed322bf hipblas images, but they didn't seem to get uploaded. I tried to build them manually with docker compose build, but it couldn't find the rocm clang.
build1.txt

The sliver of logs suggested that BUILD_TYPE was blank despite setting it in the docker-compose.yaml file... So I tried adding --build-arg BUILD_TYPE=hipblas and it appears that apt couldn't find hipblas-dev or rocblas-dev to install...
build2.txt

#2196

Cheers

[gymnae]

Hi @mudler, also wanted to try the new images, but it appears the hipblas image build failed

[bunder2015]

sha-ad5978b-hipblas-ffmpeg popped up on quay, but I'm still getting the "operator torchvision::nms does not exist" message... (still no idea why I ran into issues building the image manually)

Cheers

[mudler]

sha-ad5978b-hipblas-ffmpeg popped up on quay, but I'm still getting the "operator torchvision::nms does not exist" message... (still no idea why I ran into issues building the image manually)

Cheers

ok cool - I could finally reproduce here and #3194 should fix it. I tested manually, but need to double check after the CI builds the images as I did jumped in the container I had here and recreated the venv manually.