Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pytorch bombs if "libblas=*=*accelerate" #82

Closed
1 task done
ngam opened this issue Jan 23, 2022 · 4 comments
Closed
1 task done

pytorch bombs if "libblas=*=*accelerate" #82

ngam opened this issue Jan 23, 2022 · 4 comments

Comments

@ngam
Copy link
Contributor

ngam commented Jan 23, 2022 •
edited
Loading

Issue:

pytorch bombs if "libblas=*=*accelerate"

reference: https://conda-forge.org/docs/maintainer/knowledge_base.html#blas

reference: https://github.com/pytorch/pytorch/blob/ea4d98388500496f04fd6a814b134c4242160267/cmake/Modules/FindBLAS.cmake#L122-L135

Not sure if this part of meta makes sense:

pytorch-cpu-feedstock/recipe/meta.yaml

Lines 69 to 88 in 5711a74

- mkl-devel {{ mkl }} # [x86]
- mkl {{ mkl }} # [x86]
- libblas * *_mkl # [x86]
- libblas # [not x86]
- libblas # [not x86]
- liblapack # [not x86]
- openblas # [not x86]
- libprotobuf
- sleef
- typing
- libuv
- pkg-config # [unix]
- typing_extensions
run:
- mkl {{ mkl }} # [x86]
- libblas * *_mkl # [x86]
- libblas # [not x86]
- libcblas # [not x86]
- liblapack # [not x86]
- llvm-openmp # [osx]

Here's the error when enforcing "libblas=*=*accelerate":

Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:06) 
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/lib/python3.9/site-packages/torch/__init__.py", line 197, in <module>
    from torch._C import *  # noqa: F403
ImportError: dlopen(/Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/lib/python3.9/site-packages/torch/_C.cpython-39-darwin.so, 0x0002): Library not loaded: @rpath/libopenblas.0.dylib
  Referenced from: /Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/lib/python3.9/site-packages/torch/lib/libtorch_python.dylib
  Reason: tried: '/Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/lib/python3.9/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/lib/python3.9/site-packages/torch/lib/../../../../libopenblas.0.dylib' (no such file), '/Users/runner/miniforge3/conda-bld/pytorch-recipe_1635217266490/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libopenblas.0.dylib' (no such file), '/Users/runner/miniforge3/conda-bld/pytorch-recipe_1635217266490/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libopenblas.0.dylib' (no such file), '/Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/lib/python3.9/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/lib/python3.9/site-packages/torch/../../../libopenblas.0.dylib' (no such file), '/Users/runner/miniforge3/conda-bld/pytorch-recipe_1635217266490/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libopenblas.0.dylib' (no such file), '/Users/runner/miniforge3/conda-bld/pytorch-recipe_1635217266490/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libopenblas.0.dylib' (no such file), '/Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/lib/python3.9/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/lib/python3.9/site-packages/torch/../../../libopenblas.0.dylib' (no such file), '/Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/bin/../lib/libopenblas.0.dylib' (no such file), '/Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps/bin/../lib/libopenblas.0.dylib' (no such file), '/usr/local/lib/libopenblas.0.dylib' (no such file), '/usr/lib/libopenblas.0.dylib' (no such file)

Environment (conda list): 
$ conda list

# packages in environment at /Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps:
#
# Name                    Version                   Build  Channel
aom                       3.2.0                hc470f4d_2    conda-forge
bzip2                     1.0.8                h3422bc3_4    conda-forge
ca-certificates           2021.10.8            h4653dfc_0    conda-forge
cffi                      1.15.0           py39h52b1de0_0    conda-forge
ffmpeg                    4.4.1                hdbd4ad8_0    conda-forge
freetype                  2.10.4               h17b34a0_1    conda-forge
future                    0.18.2           py39h2804cbe_4    conda-forge
gettext                   0.19.8.1          h049c9fb_1008    conda-forge
gmp                       6.2.1                h9f76cd9_0    conda-forge
gnutls                    3.6.13               h706517b_1    conda-forge
icu                       69.1                 hbdafb3b_0    conda-forge
jbig                      2.1               h3422bc3_2003    conda-forge
jpeg                      9d                   h27ca646_0    conda-forge
lame                      3.100             h27ca646_1001    conda-forge
lcms2                     2.12                 had6a04f_0    conda-forge
lerc                      3.0                  hbdafb3b_0    conda-forge
libblas                   3.9.0           13_osxarm64_accelerate    conda-forge
libcblas                  3.9.0           13_osxarm64_accelerate    conda-forge
libcxx                    12.0.1               h168391b_1    conda-forge
libdeflate                1.8                  h3422bc3_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0.dev0      11_0_1_hf114ba7_23    conda-forge
libgfortran5              11.0.1.dev0         hf114ba7_23    conda-forge
libiconv                  1.16                 h642e427_0    conda-forge
liblapack                 3.9.0           13_osxarm64_accelerate    conda-forge
libopenblas               0.3.18          openmp_h5dd58f0_0    conda-forge
libpng                    1.6.37               hf7e6567_2    conda-forge
libprotobuf               3.18.1               hccf11d3_0    conda-forge
libtiff                   4.3.0                h74060c4_2    conda-forge
libvpx                    1.11.0               hc470f4d_3    conda-forge
libwebp-base              1.2.2                h3422bc3_1    conda-forge
libxml2                   2.9.12               hedbfbf4_1    conda-forge
libzlib                   1.2.11            hee7b306_1013    conda-forge
llvm-openmp               12.0.1               hf3c4609_1    conda-forge
lz4-c                     1.9.3                hbdafb3b_1    conda-forge
ncurses                   6.3                  hc470f4d_0    conda-forge
nettle                    3.6                  hc6a1b29_0    conda-forge
ninja                     1.10.2               hc021e02_1    conda-forge
numpy                     1.22.0           py39h61a45d2_1    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openh264                  2.1.1                habe5f53_0    conda-forge
openjpeg                  2.4.0                h062765e_1    conda-forge
openssl                   3.0.0                h3422bc3_2    conda-forge
pillow                    8.4.0            py39ha74c66e_0    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
python                    3.9.9           h43b31ca_0_cpython    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytorch                   1.10.0          cpu_py39h7601aee_0    conda-forge
pytorch-cpu               1.10.0          cpu_py39hd610c6a_0    conda-forge
readline                  8.1                  hedafd6a_0    conda-forge
setuptools                60.5.0           py39h2804cbe_0    conda-forge
sleef                     3.5.1                h156473d_2    conda-forge
sqlite                    3.37.0               h72a2b83_0    conda-forge
svt-av1                   0.9.0                hc470f4d_0    conda-forge
tk                        8.6.11               he1e0b03_1    conda-forge
torchvision               0.11.2          py39h743af5b_1_cpu    conda-forge
typing_extensions         4.0.1              pyha770c72_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
x264                      1!161.3030           h3422bc3_1    conda-forge
x265                      3.5                  h666519e_1    conda-forge
xz                        5.2.5                h642e427_1    conda-forge
zlib                      1.2.11            hee7b306_1013    conda-forge
zstd                      1.5.2                h861e0a7_0    conda-forge

Details about conda and system ( conda info ): 
$ conda info

     active environment : ptps
    active env location : /Users/ngam/.Mambaforge-MacOSX-arm64/envs/ptps
            shell level : 2
       user config file : /Users/ngam/.condarc
 populated config files : /Users/ngam/.Mambaforge-MacOSX-arm64/.condarc
                          /Users/ngam/.condarc
          conda version : 4.11.0
    conda-build version : not installed
         python version : 3.9.9.final.0
       virtual packages : __osx=12.1=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/ngam/.Mambaforge-MacOSX-arm64  (writable)
      conda av data dir : /Users/ngam/.Mambaforge-MacOSX-arm64/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/apple/osx-arm64
                          https://conda.anaconda.org/apple/noarch
                          https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/ngam/.Mambaforge-MacOSX-arm64/pkgs
                          /Users/ngam/.conda/pkgs
       envs directories : /Users/ngam/.Mambaforge-MacOSX-arm64/envs
                          /Users/ngam/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.11.0 requests/2.27.1 CPython/3.9.9 Darwin/21.2.0 OSX/12.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False
@ngam
Copy link
Contributor Author

ngam commented Jan 23, 2022

I can try to fix this in a PR unless @hmaarrfk has other thoughts/initiatives.

@hmaarrfk
Copy link
Contributor

I'm not sure if this is a pytorch issue.

This might be a libblas issue.

@ngam
Copy link
Contributor Author

ngam commented Jan 23, 2022

I'm not sure if this is a pytorch issue.

This might be a libblas issue.

No, I think it's being hard-coded here:

pytorch-cpu-feedstock/recipe/build_pytorch.sh

Line 75 in 5711a74

export BLAS=OpenBLAS

and here:

pytorch-cpu-feedstock/recipe/meta.yaml

Line 75 in 5711a74

- openblas # [not x86]

@hmaarrfk
Copy link
Contributor

i'm a little confused about the necessary changes, but if you have them, then go for it. We can build OSX on the CIs quickly.

This was referenced Jan 23, 2022
Closed
Closed
Closed
@ngam ngam mentioned this issue Feb 4, 2022
Closed
5 tasks
@ngam ngam closed this as completed Feb 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

implement accelerate for osx-arm64 ngam/pytorch-cpu-feedstock
2 participants
@hmaarrfk @ngam