[Build] /usr/bin/ld: failed to set dynamic section sizes: memory exhausted #15337

goldyfruit · 2023-04-03T17:25:58Z

Describe the issue

Unable to build onnxruntime on Debian Bookworm container for armv7.

Urgency

No response

Target platform

armv7

Build script

Dockerfile-debian.arm32v7

FROM arm32v7/debian:bookworm
ADD . /code

RUN /code/dockerfiles/scripts/install_debian_armv7.sh && cd /code && ./build.sh --parallel --build_wheel --skip_test --allow_running_as_root --skip_submodule_sync --update --build --cmake_extra_defines ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER)

install_debian_armv7.sh

apt-get update
apt-get install -y python3-pip python3-wheel python3-dev python3-numpy python3-packaging git build-essential gcc g++ git cmake protobuf-compiler libprotoc-dev python3-setuptools

cd onnxruntime
docker build -t onnxruntime-source -f dockerfiles/Dockerfile-debian.arm32v7 .
```

### Error / output

```#0 7614.1       |   ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#0 7614.1 /usr/bin/ld: warning: QgemmU8X8KernelNeon.S.o: missing .note.GNU-stack section implies executable stack
#0 7614.1 /usr/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
#0 7614.1 /usr/bin/ld: failed to set dynamic section sizes: memory exhausted
#0 7614.1 collect2: error: ld returned 1 exit status
#0 7614.1 gmake[2]: *** [CMakeFiles/onnxruntime_test_all.dir/build.make:4768: onnxruntime_test_all] Error 1
#0 7614.1 gmake[1]: *** [CMakeFiles/Makefile2:2130: CMakeFiles/onnxruntime_test_all.dir/all] Error 2
#0 7614.1 gmake: *** [Makefile:166: all] Error 2
#0 7614.1 Traceback (most recent call last):
#0 7614.1   File "/code/tools/ci_build/build.py", line 2797, in <module>
#0 7614.1     sys.exit(main())
#0 7614.1              ^^^^^^
#0 7614.1   File "/code/tools/ci_build/build.py", line 2693, in main
#0 7614.1     build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
#0 7614.1   File "/code/tools/ci_build/build.py", line 1444, in build_targets
#0 7614.1     run_subprocess(cmd_args, env=env)
#0 7614.1   File "/code/tools/ci_build/build.py", line 785, in run_subprocess
#0 7614.1     return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
#0 7614.1            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 7614.1   File "/code/tools/python/util/run.py", line 49, in run
#0 7614.1     completed_process = subprocess.run(
#0 7614.1                         ^^^^^^^^^^^^^^^
#0 7614.1   File "/usr/lib/python3.11/subprocess.py", line 571, in run
#0 7614.1     raise CalledProcessError(retcode, process.args,
#0 7614.1 subprocess.CalledProcessError: Command '['/usr/bin/cmake', '--build', '/code/build/Linux/Debug', '--config', 'Debug', '--', '-j32']' returned non-zero exit status 2.
```

### Visual Studio Version

_No response_

### GCC / Compiler Version

12.2.0

The text was updated successfully, but these errors were encountered:

baijumeswani · 2023-04-04T16:24:53Z

From the linker error message, it seems like the machine doesn't have enough ram to link the code.

I am not sure if there is a solution to this problem other than increasing machine ram. Maybe @snnn has some helpful insight for this?

goldyfruit · 2023-04-04T16:54:36Z

@baijumeswani thanks for your answer, I got 256GB of RAM total on the server and 182GB are free. I was able as well to build the aarch64 version on the same server.

Maybe it's related to a 4GB limit, don't really know.

baijumeswani · 2023-04-04T21:00:11Z

@goldyfruit thanks for the info. I should not have assumed. :)

I'm sorry but I have no good reason why you're seeing the linker error. I'll try to consult with other engineers internally, but not sure if I will have any luck.

If you do find the answer to the problem, please post it here so it can benefit others who might find themselves in a similar boat.

goldyfruit · 2023-04-04T21:48:31Z

@goldyfruit thanks for the info. I should not have assumed. :)

You couldn't 👍

I'm sorry but I have no good reason why you're seeing the linker error. I'll try to consult with other engineers internally, but not sure if I will have any luck.

I hope you will have some update.

If you do find the answer to the problem, please post it here so it can benefit others who might find themselves in a similar boat.

Sure.

satyajandhyala · 2023-05-05T22:05:59Z

@goldyfruit Did you try removing "--parallel" build.sh command-line option?

goldyfruit · 2023-05-05T22:17:48Z

@goldyfruit Did you try removing "--parallel" build.sh command-line option?

No I don't, let me try without it!

goldyfruit · 2023-05-06T11:33:42Z

Still running... 13 hours!

goldyfruit · 2023-05-07T12:39:04Z

Same error after 27 hours of build.

#0 99401.8 /usr/bin/ld: warning: QgemmU8X8KernelNeon.S.o: missing .note.GNU-stack section implies executable stack
#0 99401.8 /usr/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
#0 99401.8 /usr/bin/ld: failed to set dynamic section sizes: memory exhausted
#0 99401.8 collect2: error: ld returned 1 exit status
#0 99401.8 gmake[2]: *** [CMakeFiles/onnxruntime_test_all.dir/build.make:4768: onnxruntime_test_all] Error 1
#0 99401.8 gmake[1]: *** [CMakeFiles/Makefile2:2130: CMakeFiles/onnxruntime_test_all.dir/all] Error 2
#0 99401.8 gmake: *** [Makefile:166: all] Error 2
#0 99401.8 Traceback (most recent call last):
#0 99401.8   File "/code/tools/ci_build/build.py", line 2773, in <module>
#0 99401.8     sys.exit(main())
#0 99401.8              ^^^^^^
#0 99401.8   File "/code/tools/ci_build/build.py", line 2669, in main
#0 99401.8     build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
#0 99401.8   File "/code/tools/ci_build/build.py", line 1447, in build_targets
#0 99401.8     run_subprocess(cmd_args, env=env)
#0 99401.8   File "/code/tools/ci_build/build.py", line 787, in run_subprocess
#0 99401.8     return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
#0 99401.8            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 99401.8   File "/code/tools/python/util/run.py", line 49, in run
#0 99401.8     completed_process = subprocess.run(
#0 99401.8                         ^^^^^^^^^^^^^^^
#0 99401.8   File "/usr/lib/python3.11/subprocess.py", line 571, in run
#0 99401.8     raise CalledProcessError(retcode, process.args,
#0 99401.8 subprocess.CalledProcessError: Command '['/usr/bin/cmake', '--build', '/code/build/Linux/Debug', '--config', 'Debug']' returned non-zero exit status 2.

satyajandhyala · 2023-05-08T22:13:31Z

Can you try building a Release version instead of a Debug version? I am hoping that Release build might succeed. You can specify the release build by adding " --config Release" option to the build.sh command.

satyajandhyala · 2023-05-11T16:50:18Z

@goldyfruit Did you try building Release build?

goldyfruit · 2023-05-11T17:01:55Z

@goldyfruit Did you try building Release build?

Build in progress (thanks for the reminder 👍 )

goldyfruit · 2023-05-12T00:25:21Z

Did a git pull and now it requires Cmake 3.26...

#0 3828.7 CMake Error at CMakeLists.txt:11 (cmake_minimum_required):
#0 3828.7   CMake 3.26 or higher is required.  You are running version 3.25.1
#0 3828.7 
#0 3828.7 
#0 3828.7 -- Configuring incomplete, errors occurred!
#0 3828.7 Traceback (most recent call last):
#0 3828.7   File "/code/tools/ci_build/build.py", line 2601, in <module>
#0 3828.7     sys.exit(main())
#0 3828.7              ^^^^^^
#0 3828.7   File "/code/tools/ci_build/build.py", line 2462, in main
#0 3828.7     generate_build_tree(
#0 3828.7   File "/code/tools/ci_build/build.py", line 1371, in generate_build_tree
#0 3828.7     run_subprocess(
#0 3828.7   File "/code/tools/ci_build/build.py", line 780, in run_subprocess
#0 3828.7     return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
#0 3828.7            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 3828.7   File "/code/tools/python/util/run.py", line 49, in run
#0 3828.7     completed_process = subprocess.run(
#0 3828.7                         ^^^^^^^^^^^^^^^
#0 3828.7   File "/usr/lib/python3.11/subprocess.py", line 571, in run
#0 3828.7     raise CalledProcessError(retcode, process.args,
#0 3828.7 subprocess.CalledProcessError: Command '['/usr/bin/cmake', '/code/cmake', '-Donnxruntime_RUN_ONNX_TESTS=OFF', '-Donnxruntime_GENERATE_TEST_REPORTS=ON', '-DPython_EXECUTABLE=/usr/bin/python3', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-Donnxruntime_USE_MIMALLOC=OFF', '-Donnxruntime_ENABLE_PYTHON=ON', '-Donnxruntime_BUILD_CSHARP=OFF', '-Donnxruntime_BUILD_JAVA=OFF', '-Donnxruntime_BUILD_NODEJS=OFF', '-Donnxruntime_BUILD_OBJC=OFF', '-Donnxruntime_BUILD_SHARED_LIB=OFF', '-Donnxruntime_BUILD_APPLE_FRAMEWORK=OFF', '-Donnxruntime_USE_DNNL=OFF', '-Donnxruntime_USE_NNAPI_BUILTIN=OFF', '-Donnxruntime_USE_RKNPU=OFF', '-Donnxruntime_USE_LLVM=OFF', '-Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF', '-Donnxruntime_USE_VITISAI=OFF', '-Donnxruntime_USE_TENSORRT=OFF', '-Donnxruntime_USE_TENSORRT_BUILTIN_PARSER=ON', '-Donnxruntime_USE_TVM=OFF', '-Donnxruntime_TVM_CUDA_RUNTIME=OFF', '-Donnxruntime_TVM_USE_HASH=OFF', '-Donnxruntime_USE_MIGRAPHX=OFF', '-Donnxruntime_DISABLE_CONTRIB_OPS=OFF', '-Donnxruntime_DISABLE_ML_OPS=OFF', '-Donnxruntime_DISABLE_RTTI=OFF', '-Donnxruntime_DISABLE_EXCEPTIONS=OFF', '-Donnxruntime_MINIMAL_BUILD=OFF', '-Donnxruntime_EXTENDED_MINIMAL_BUILD=OFF', '-Donnxruntime_MINIMAL_BUILD_CUSTOM_OPS=OFF', '-Donnxruntime_REDUCED_OPS_BUILD=OFF', '-Donnxruntime_ENABLE_LANGUAGE_INTEROP_OPS=OFF', '-Donnxruntime_USE_DML=OFF', '-Donnxruntime_USE_WINML=OFF', '-Donnxruntime_BUILD_MS_EXPERIMENTAL_OPS=OFF', '-Donnxruntime_USE_TELEMETRY=OFF', '-Donnxruntime_ENABLE_LTO=OFF', '-Donnxruntime_USE_ACL=OFF', '-Donnxruntime_USE_ACL_1902=OFF', '-Donnxruntime_USE_ACL_1905=OFF', '-Donnxruntime_USE_ACL_1908=OFF', '-Donnxruntime_USE_ACL_2002=OFF', '-Donnxruntime_USE_ARMNN=OFF', '-Donnxruntime_ARMNN_RELU_USE_CPU=ON', '-Donnxruntime_ARMNN_BN_USE_CPU=ON', '-Donnxruntime_USE_JSEP=OFF', '-Donnxruntime_ENABLE_NVTX_PROFILE=OFF', '-Donnxruntime_ENABLE_TRAINING=OFF', '-Donnxruntime_ENABLE_TRAINING_OPS=OFF', '-Donnxruntime_ENABLE_TRAINING_APIS=OFF', '-Donnxruntime_ENABLE_CPU_FP16_OPS=OFF', '-Donnxruntime_USE_NCCL=OFF', '-Donnxruntime_BUILD_BENCHMARKS=OFF', '-Donnxruntime_USE_ROCM=OFF', '-DOnnxruntime_GCOV_COVERAGE=OFF', '-Donnxruntime_USE_MPI=OFF', '-Donnxruntime_ENABLE_MEMORY_PROFILE=OFF', '-Donnxruntime_ENABLE_CUDA_LINE_NUMBER_INFO=OFF', '-Donnxruntime_BUILD_WEBASSEMBLY=OFF', '-Donnxruntime_BUILD_WEBASSEMBLY_STATIC_LIB=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_CATCHING=ON', '-Donnxruntime_ENABLE_WEBASSEMBLY_API_EXCEPTION_CATCHING=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_THROWING=ON', '-Donnxruntime_WEBASSEMBLY_RUN_TESTS_IN_BROWSER=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_THREADS=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_DEBUG_INFO=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_PROFILING=OFF', '-Donnxruntime_ENABLE_LAZY_TENSOR=OFF', '-Donnxruntime_ENABLE_EXTERNAL_CUSTOM_OP_SCHEMAS=OFF', '-Donnxruntime_ENABLE_CUDA_PROFILING=OFF', '-Donnxruntime_ENABLE_ROCM_PROFILING=OFF', '-Donnxruntime_USE_XNNPACK=OFF', '-Donnxruntime_USE_WEBNN=OFF', '-Donnxruntime_USE_CANN=OFF', '-DONNXRUNTIME_VERSION=1.16.0', '-DCMAKE_TLS_VERIFY=ON', '-DFETCHCONTENT_QUIET=OFF', '-Donnxruntime_ENABLE_MEMLEAK_CHECKER=OFF', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_PREFIX_PATH=/code/build/Linux/Release/installed']' returned non-zero exit status 1.
------

I give up for now, thanks for helping.

satyajandhyala · 2023-05-12T00:42:37Z

When you get back to this, you can install/upgrade cmake using "pip install cmake"

vlasov01 · 2024-10-23T02:22:57Z

I've got the same issue while building for Cortex A9 (armv7l).

gmake[2]: *** [CMakeFiles/onnxruntime_pybind11_state.dir/build.make:205: CMakeFiles/onnxruntime_pybind11_state.dir/home/xilinx/onnxruntime/onnxruntime/python/onnxruntime_pybind_state.cc.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:3127: CMakeFiles/onnxruntime_pybind11_state.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: failed to set dynamic section sizes: memory exhausted
collect2: error: ld returned 1 exit status
gmake[2]: *** [CMakeFiles/onnxruntime_test_all.dir/build.make:5243: onnxruntime_test_all] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:3334: CMakeFiles/onnxruntime_test_all.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2
Traceback (most recent call last):
  File "/home/xilinx/onnxruntime/tools/ci_build/build.py", line 2998, in <module>
    sys.exit(main())
  File "/home/xilinx/onnxruntime/tools/ci_build/build.py", line 2888, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/home/xilinx/onnxruntime/tools/ci_build/build.py", line 1739, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/home/xilinx/onnxruntime/tools/ci_build/build.py", line 867, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
  File "/home/xilinx/onnxruntime/tools/python/util/run.py", line 49, in run
    completed_process = subprocess.run(
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/bin/cmake', '--build', '/home/xilinx/onnxruntime/build/Linux/Release', '--config', 'Release', '--', '-j2']' returned non-zero exit status 2.

OS
Linux pynq 5.15.19-xilinx-v2022.1 #1 SMP PREEMPT Mon Apr 11 17:52:14 UTC 2022 armv7l armv7l armv7l GNU/Linux

GNU ld (GNU Binutils for Ubuntu) 2.38
gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
cmake version 3.31.20241016-gee2eadb

./build.sh --config Release --parallel --skip_submodule_sync --skip_tests --use_xnnpack --build_wheel --cmake_extra_defines CMAKE_CXX_FLAGS="-Wno-dev -Wall -Wextra -O3 -g3 -mfloat-abi=hard -mfpu=neon-vfpv3 -mcpu=cortex-a9 -mfloat-abi=hard" CMAKE_C_FLAGS="-Wno-dev -Wall -Wextra -O3 -g3 -mfloat-abi=hard -mfpu=neon-vfpv3 -mcpu=cortex-a9 -mfloat-abi=hard"

During the last failed build I had 16GB of swap. Now I've doubled it.

#free
               total        used        free      shared  buff/cache   available
Mem:          505428      474592       10144          36       20692       19152
Swap:       33554428     2553600    31000828

goldyfruit added the build build issues; typically submitted using template label Apr 3, 2023

goldyfruit closed this as completed May 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Build] /usr/bin/ld: failed to set dynamic section sizes: memory exhausted #15337

[Build] /usr/bin/ld: failed to set dynamic section sizes: memory exhausted #15337

goldyfruit commented Apr 3, 2023

baijumeswani commented Apr 4, 2023

goldyfruit commented Apr 4, 2023 •

edited

Loading

baijumeswani commented Apr 4, 2023

goldyfruit commented Apr 4, 2023

satyajandhyala commented May 5, 2023

goldyfruit commented May 5, 2023

goldyfruit commented May 6, 2023

goldyfruit commented May 7, 2023

satyajandhyala commented May 8, 2023 •

edited

Loading

satyajandhyala commented May 11, 2023

goldyfruit commented May 11, 2023

goldyfruit commented May 12, 2023 •

edited

Loading

satyajandhyala commented May 12, 2023

vlasov01 commented Oct 23, 2024

[Build] /usr/bin/ld: failed to set dynamic section sizes: memory exhausted #15337

[Build] /usr/bin/ld: failed to set dynamic section sizes: memory exhausted #15337

Comments

goldyfruit commented Apr 3, 2023

Describe the issue

Urgency

Target platform

Build script

baijumeswani commented Apr 4, 2023

goldyfruit commented Apr 4, 2023 • edited Loading

baijumeswani commented Apr 4, 2023

goldyfruit commented Apr 4, 2023

satyajandhyala commented May 5, 2023

goldyfruit commented May 5, 2023

goldyfruit commented May 6, 2023

goldyfruit commented May 7, 2023

satyajandhyala commented May 8, 2023 • edited Loading

satyajandhyala commented May 11, 2023

goldyfruit commented May 11, 2023

goldyfruit commented May 12, 2023 • edited Loading

satyajandhyala commented May 12, 2023

vlasov01 commented Oct 23, 2024

goldyfruit commented Apr 4, 2023 •

edited

Loading

satyajandhyala commented May 8, 2023 •

edited

Loading

goldyfruit commented May 12, 2023 •

edited

Loading