[Build] Cross build failure for CPU architecture ARM Cortex A53 with Flags "-march=armv8-a -mtune=cortex-a53" due to Eigen3 library. #22463

shadab205 · 2024-10-16T07:14:27Z

Describe the issue

I am trying to build onnxruntime for a Raspberry Pi 3 with the CPU core architecture Cortex-A53 but the build fails and the issue are pointing to Eigen3 Library.

I am cross building it on a WSL Ubuntu 22.04 setup on a Windows machine. But I also tried to build it natively on the Raspberry Pi with similar failures.

Urgency

The build failure is blocking a project from moving forward.

Target platform

Linux aarch64

Build script

cmake .. -Donnxruntime_ENABLE_CPUINFO=OFF -Donnxruntime_GCC_STATIC_CPP_RUNTIME=ON -DCMAKE_C_FLAGS="-march=armv8-a -mtune=cortex-a53 -Wno-error" -DCMAKE_CXX_FLAGS="-march=armv8-a -mtune=cortex-a53 -Wno-error" Donnxruntime_DEV_MODE=OFF -DCMAKE_TOOLCHAIN_FILE=arm64.toolchain.cmake -DCMAKE_BUILD_TYPE=Release -Donnxruntime_BUILD_UNIT_TESTS=OFF -Donnxruntime_BUILD_SHARED_LIB=ON -Donnxruntime_USE_CUDA=OFF -Donnxruntime_USE_DML=OFF

Custom Toolchain File:

Specify the cross compiler

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

Specify the cross compiler

set(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
set(CMAKE_FIND_ROOT_PATH /usr/aarch64-linux-gnu)

Search for programs in the build host directories

set(CMAKE_FIND_PROGRAMS)

Search for headers and libraries in the target environment

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

Error / output

/home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:855:85: error: type/value mismatch at argument 1 in template parameter list for ‘template const Eigen::CwiseBinaryOp<Eigen::internal::scalar_max_op<typename Eigen::internal::traits::Scalar, typename Eigen::internal::traits::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase::max(const Eigen::ArrayBase&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
855 | per_iter_bh.EigenInput0().array().template maxEigen::PropagateNaN(per_iter_bh.ScalarInput1());
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:855:85: note: expected a type, got ‘Eigen::PropagateNaN’
/home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:859:85: error: no matching function for call to ‘Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >::maxEigen::PropagateNaN(Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >)’
859 | per_iter_bh.EigenInput0().array().template maxEigen::PropagateNaN(
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
860 | per_iter_bh.EigenInput1().array());
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/eigen3/Eigen/Core:19,
from /home/user/onnxruntime/onnxruntime/core/util/math_cpuonly.h:68,
from /home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.h:10,
from /home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:4:
/usr/include/eigen3/Eigen/src/Core/../plugins/ArrayCwiseBinaryOps.h:59:28: note: candidate: ‘template const Eigen::CwiseBinaryOp<Eigen::internal::scalar_max_op<typename Eigen::internal::traits::Scalar, typename Eigen::internal::traits::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase::max(const Eigen::ArrayBase&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
59 | EIGEN_MAKE_CWISE_BINARY_OP(max,max)
| ^~~
/usr/include/eigen3/Eigen/src/Core/util/Macros.h:1339:4: note: in definition of macro ‘EIGEN_MAKE_CWISE_BINARY_OP’
1339 | (METHOD)(const EIGEN_CURRENT_STORAGE_BASE_CLASS &other) const
| ^~~~~~
/usr/include/eigen3/Eigen/src/Core/../plugins/ArrayCwiseBinaryOps.h:59:28: note: template argument deduction/substitution failed:
59 | EIGEN_MAKE_CWISE_BINARY_OP(max,max)
| ^~~
/usr/include/eigen3/Eigen/src/Core/util/Macros.h:1339:4: note: in definition of macro ‘EIGEN_MAKE_CWISE_BINARY_OP’
1339 | (METHOD)(const EIGEN_CURRENT_STORAGE_BASE_CLASS &other) const
| ^~~~~~
/home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:859:85: error: type/value mismatch at argument 1 in template parameter list for ‘template const Eigen::CwiseBinaryOp<Eigen::internal::scalar_max_op<typename Eigen::internal::traits::Scalar, typename Eigen::internal::traits::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase::max(const Eigen::ArrayBase&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
859 | per_iter_bh.EigenInput0().array().template maxEigen::PropagateNaN(
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
860 | per_iter_bh.EigenInput1().array());
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:859:85: note: expected a type, got ‘Eigen::PropagateNaN’

Visual Studio Version

No response

GCC / Compiler Version

aarch64-linux-gnu v11.4.0

vlasov01 · 2024-10-19T11:08:20Z

I have the same error as well. But I'm building it directly on the ARM Cortex-A9 device (Linux pynq 5.15.19-xilinx-v2022.1 #1 SMP PREEMPT Mon Apr 11 17:52:14 UTC 2022 armv7l armv7l armv7l GNU/Linux ) using cmake version 3.31.20241016-gee2eadb.

Here is my build command:
./build.sh --config Release --parallel --skip_submodule_sync --skip_tests --use_xnnpack --build_wheel --cmake_extra_defines CMAKE_CXX_FLAGS="-Wno-dev -Wall -Wextra -O3 -g3 -mfloat-abi=hard -mfpu=neon-vfpv3 -mcpu=cortex-a9 -mfloat-abi=hard" CMAKE_C_FLAGS="-Wno-dev -Wall -Wextra -O3 -g3 -mfloat-abi=hard -mfpu=neon-vfpv3 -mcpu=cortex-a9 -mfloat-abi=hard"

sevagh · 2024-10-20T16:13:42Z

I'm having the same issue building for Linux x86_64 on an x86_64 computer:

/home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc: In member function ‘onnxruntime::common::Status onnxruntime::Min_6<T>::Compute(onnxruntime::OpKernelContext*) const [with T = float]’:
/home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:708:56: error: no matching function for call to ‘Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1>, 0, Eigen::Stride<0, 0> > >::min<Eigen::PropagateNaN>(Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<float, -1, 1>, 0, Eigen::Stride<0, 0> > >)’
  708 |     min = min.array().template min<Eigen::PropagateNaN>(EigenMap<float>(data_n).array());
      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/eigen3/Eigen/Core:19,
                 from /home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/util/math_cpuonly.h:68,
                 from /home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.h:10,
                 from /home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:4:
/usr/include/eigen3/Eigen/src/Core/../plugins/ArrayCwiseBinaryOps.h:33:28: note: candidate: ‘template<class OtherDerived> const Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<typename Eigen::internal::traits<T>::Scalar, typename Eigen::internal::traits<OtherDerived>::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase<Derived>::min(const Eigen::ArrayBase<OtherDerived>&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
   33 | EIGEN_MAKE_CWISE_BINARY_OP(min,min)
      |                            ^~~
/usr/include/eigen3/Eigen/src/Core/util/Macros.h:1339:4: note: in definition of macro ‘EIGEN_MAKE_CWISE_BINARY_OP’
 1339 |   (METHOD)(const EIGEN_CURRENT_STORAGE_BASE_CLASS<OtherDerived> &other) const \
      |    ^~~~~~
/usr/include/eigen3/Eigen/src/Core/../plugins/ArrayCwiseBinaryOps.h:33:28: note:   template argument deduction/substitution failed:
   33 | EIGEN_MAKE_CWISE_BINARY_OP(min,min)
      |                            ^~~
/usr/include/eigen3/Eigen/src/Core/util/Macros.h:1339:4: note: in definition of macro ‘EIGEN_MAKE_CWISE_BINARY_OP’
 1339 |   (METHOD)(const EIGEN_CURRENT_STORAGE_BASE_CLASS<OtherDerived> &other) const \
      |    ^~~~~~
/home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:708:56: error: type/value mismatch at argument 1 in template parameter list for ‘template<class OtherDerived> const Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<typename Eigen::internal::traits<T>::Scalar, typename Eigen::internal::traits<OtherDerived>::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase<Derived>::min(const Eigen::ArrayBase<OtherDerived>&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
  708 |     min = min.array().template min<Eigen::PropagateNaN>(EigenMap<float>(data_n).array());
      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:708:56: note:   expected a type, got ‘Eigen::PropagateNaN’

sevagh · 2024-10-20T17:05:23Z

What version of Eigen are you using? I was able to bypass this issue by pointing Onnxruntime to use a local copy of Eigen which is newer than my system Eigen3.

The 3.4 branch doesn't have it, you need to use Eigen from master.

From your original log:
/usr/include/eigen3/Eigen/

Seems to me you installed Eigen from your system package manager, which is the same as mine (and those are generally oudated).

However, pointing to your own custom path of Eigen with the build options is a bit tricky: #22515 - it will default to your system's Eigen even if you supply --eigen_path=/path/to/newer/eigen. I commented out the find_package call to force it to obey my supplied eigen_path:

--- a/cmake/external/onnxruntime_external_deps.cmake
+++ b/cmake/external/onnxruntime_external_deps.cmake
@@ -543,12 +543,12 @@ if(TARGET ONNX::onnx_proto AND NOT TARGET onnx_proto)
   add_library(onnx_proto ALIAS ONNX::onnx_proto)
 endif()

-find_package(Eigen3 CONFIG)
-if(Eigen3_FOUND)
-  get_target_property(eigen_INCLUDE_DIRS Eigen3::Eigen INTERFACE_INCLUDE_DIRECTORIES)
-else()
-  include(eigen) # FetchContent
-endif()
+#find_package(Eigen3 CONFIG)
+#if(Eigen3_FOUND)
+#  get_target_property(eigen_INCLUDE_DIRS Eigen3::Eigen INTERFACE_INCLUDE_DIRECTORIES)
+#else()
+include(eigen) # FetchContent
+#endif()

vlasov01 · 2024-10-20T19:13:36Z

It seems I was able to bypass this issue by removing my system Eigen3. I'm not 100% sure as the build is still in progress (91%).

sudo apt remove libeigen3-dev

sevagh · 2024-10-20T19:17:44Z

Makes sense to me - if you delete your outdated system Eigen3, onnxruntime will download a newer version which supports this template

ep1cman · 2024-10-20T19:18:23Z

I encountered this issues too. It turns out the cmake scripts are a bit flawed. If egien is found on the system it will always use that. It will never import the cmake file and thus it will not honour the --use_preinstalled_eigen flag not being set. Not really sure what the intended purpose of this flag is but currently, if there is a system provided eigen, it will always use that.

To get the build work for me I just commented out some lines in the cmake such that it always runs the eigen cmake file and downloads its own copy.

sevagh · 2024-10-20T19:36:48Z

I encountered this issues too. It turns out the cmake scripts are a bit flawed. If egien is found on the system it will always use that. It will never import the cmake file and thus it will not honour the --use_preinstalled_eigen flag not being set. Not really sure what the intended purpose of this flag is but currently, if there is a system provided eigen, it will always use that.

To get the build work for me I just commented out some lines in the cmake such that it always runs the eigen cmake file and downloads its own copy.

FYI, I made an issue for this: #22515

vlasov01 · 2024-10-21T02:02:19Z

Makes sense to me - if you delete your outdated system Eigen3, onnxruntime will download a newer version which supports this template

Actually, I had the latest release of libeigen3-dev 3.4.0-2ubuntu2.
As I understand onnxruntime is using a release candidate of it as per #18286.

shadab205 · 2024-10-21T07:14:00Z

Makes sense to me - if you delete your outdated system Eigen3, onnxruntime will download a newer version which supports this template

Actually, I had the latest release of libeigen3-dev 3.4.0-2ubuntu2. As I understand onnxruntime is using a release candidate of it as per #18286.

I have 3.4.0-2ubuntu2 version installed as well. I will try to remove the system package and try again.

shadab205 · 2024-10-21T07:15:44Z

I encountered this issues too. It turns out the cmake scripts are a bit flawed. If egien is found on the system it will always use that. It will never import the cmake file and thus it will not honour the --use_preinstalled_eigen flag not being set. Not really sure what the intended purpose of this flag is but currently, if there is a system provided eigen, it will always use that.

To get the build work for me I just commented out some lines in the cmake such that it always runs the eigen cmake file and downloads its own copy.

@ep1cman Can you tell us what changes did you do to the cmake files? Is it the same as suggested by @sevagh above

snnn · 2024-10-22T02:40:14Z

Does it latest eigen version work? You may open cmake/deps.txt then search the eigen line and update it. The last column is the SHA1 sum of the source zip file.

If it works. I will submit a PR to update it.

shadab205 · 2024-10-22T07:02:10Z

I will try to compile it again.

shadab205 · 2024-10-22T08:19:58Z

Does it latest eigen version work? You may open cmake/deps.txt then search the eigen line and update it. The last column is the SHA1 sum of the source zip file.

If it works. I will submit a PR to update it.

@snnn I can confirm that the issue is resolved( and as a plus the issue with the nsync has been resolved). I will close this issue. Thanks.

sevagh · 2024-10-29T19:19:03Z

I don't think you should close the issue, given that we all needed to use workarounds. The onnxruntime project itself needs to merge a fix first, right?

snnn · 2024-10-29T20:12:46Z

Would the newly added vcpkg feature work for you? #21348

sevagh · 2024-10-29T21:16:55Z

I am on Linux so I get an issue with WIL (windows implementation libraries)?

if(onnxruntime_USE_VCPKG)
  find_package(wil CONFIG REQUIRED)
  set(WIL_TARGET "WIL::WIL")
else()
  include(wil) # FetchContent
endif()

sevagh · 2024-10-29T21:22:52Z

Finally I settled for removing my system Eigen3, which will make "use_preinstalled_eigen" work correctly 🤷

shadab205 added the build build issues; typically submitted using template label Oct 16, 2024

shadab205 closed this as completed Oct 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Build] Cross build failure for CPU architecture ARM Cortex A53 with Flags "-march=armv8-a -mtune=cortex-a53" due to Eigen3 library. #22463

[Build] Cross build failure for CPU architecture ARM Cortex A53 with Flags "-march=armv8-a -mtune=cortex-a53" due to Eigen3 library. #22463

shadab205 commented Oct 16, 2024

vlasov01 commented Oct 19, 2024 •

edited

Loading

sevagh commented Oct 20, 2024

sevagh commented Oct 20, 2024 •

edited

Loading

vlasov01 commented Oct 20, 2024 •

edited

Loading

sevagh commented Oct 20, 2024

ep1cman commented Oct 20, 2024 •

edited

Loading

sevagh commented Oct 20, 2024

vlasov01 commented Oct 21, 2024

shadab205 commented Oct 21, 2024

shadab205 commented Oct 21, 2024

snnn commented Oct 22, 2024

shadab205 commented Oct 22, 2024

shadab205 commented Oct 22, 2024

sevagh commented Oct 29, 2024

snnn commented Oct 29, 2024

sevagh commented Oct 29, 2024

sevagh commented Oct 29, 2024

[Build] Cross build failure for CPU architecture ARM Cortex A53 with Flags "-march=armv8-a -mtune=cortex-a53" due to Eigen3 library. #22463

[Build] Cross build failure for CPU architecture ARM Cortex A53 with Flags "-march=armv8-a -mtune=cortex-a53" due to Eigen3 library. #22463

Comments

shadab205 commented Oct 16, 2024

Describe the issue

Urgency

Target platform

Build script

Specify the cross compiler

Specify the cross compiler

Search for programs in the build host directories

Search for headers and libraries in the target environment

Error / output

Visual Studio Version

GCC / Compiler Version

vlasov01 commented Oct 19, 2024 • edited Loading

sevagh commented Oct 20, 2024

sevagh commented Oct 20, 2024 • edited Loading

vlasov01 commented Oct 20, 2024 • edited Loading

sevagh commented Oct 20, 2024

ep1cman commented Oct 20, 2024 • edited Loading

sevagh commented Oct 20, 2024

vlasov01 commented Oct 21, 2024

shadab205 commented Oct 21, 2024

shadab205 commented Oct 21, 2024

snnn commented Oct 22, 2024

shadab205 commented Oct 22, 2024

shadab205 commented Oct 22, 2024

sevagh commented Oct 29, 2024

snnn commented Oct 29, 2024

sevagh commented Oct 29, 2024

sevagh commented Oct 29, 2024

vlasov01 commented Oct 19, 2024 •

edited

Loading

sevagh commented Oct 20, 2024 •

edited

Loading

vlasov01 commented Oct 20, 2024 •

edited

Loading

ep1cman commented Oct 20, 2024 •

edited

Loading