Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] ci_cpu docker image build fail #17070

Closed
cissoidx opened this issue Jun 7, 2024 · 6 comments · Fixed by #17337
Closed

[Bug] ci_cpu docker image build fail #17070

cissoidx opened this issue Jun 7, 2024 · 6 comments · Fixed by #17337
Labels
needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it type: bug

Comments

@cissoidx
Copy link

cissoidx commented Jun 7, 2024

Expected behavior

The docker image (ci_cpu) builds successfully.

Actual behavior

failed with the following error

40.25 [ 69%] Built target nnpack_reference_layers
40.88 [ 71%] Generating src/x86_64-fma/2d-fourier-16x16.py.o
43.07 [ 72%] Generating src/x86_64-fma/2d-winograd-8x8-3x3.py.o
43.44 [ 73%] Generating src/x86_64-fma/blas/s8gemm.py.o
43.56 [ 74%] Generating src/x86_64-fma/blas/c8gemm.py.o
43.76 [ 75%] Generating src/x86_64-fma/blas/s4c6gemm.py.o
43.94 [ 77%] Generating src/x86_64-fma/blas/conv1x1.py.o
44.04 [ 78%] Generating src/x86_64-fma/blas/sgemm.py.o
44.18 [ 79%] Generating src/x86_64-fma/max-pooling.py.o
44.25 [ 80%] Generating src/x86_64-fma/relu.py.o
44.30 [ 81%] Generating src/x86_64-fma/softmax.py.o
44.57 [ 83%] Generating src/x86_64-fma/blas/sdotxf.py.o
44.78 [ 84%] Generating src/x86_64-fma/blas/shdotxf.py.o
44.81 Traceback (most recent call last):
44.81   File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
44.81     return _run_code(code, main_globals, None,
44.81   File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
44.81     exec(code, run_globals)
44.81   File "/NNPACK/deps/peachpy/peachpy/x86_64/__main__.py", line 282, in <module>
44.81     main()
44.81   File "/NNPACK/deps/peachpy/peachpy/x86_64/__main__.py", line 269, in main
44.81     execute_script(writers, options.input[0])
44.81   File "/NNPACK/deps/peachpy/peachpy/x86_64/__main__.py", line 200, in execute_script
44.81     execute_script(writers, source_filename)
44.81   File "/NNPACK/deps/peachpy/peachpy/x86_64/__main__.py", line 200, in execute_script
44.81     execute_script(writers, source_filename)
44.81   File "/NNPACK/deps/peachpy/peachpy/x86_64/__main__.py", line 204, in execute_script
44.81     exec(code, globals())
44.81   File "/NNPACK/src/x86_64-fma/blas/shdotxf.py", line 4, in <module>
44.81     from fp16.avx import fp16_alt_xmm_to_fp32_xmm
44.81 ModuleNotFoundError: No module named 'fp16.avx'
44.81 make[2]: *** [CMakeFiles/nnpack.dir/build.make:188: src/x86_64-fma/blas/shdotxf.py.o] Error 1
44.81 make[1]: *** [CMakeFiles/Makefile2:175: CMakeFiles/nnpack.dir/all] Error 2
44.81 make: *** [Makefile:136: all] Error 2
------
Dockerfile.ci_cpu:82
--------------------
  80 |     # NNPACK deps
  81 |     COPY install/ubuntu_install_nnpack.sh /install/ubuntu_install_nnpack.sh
  82 | >>> RUN bash /install/ubuntu_install_nnpack.sh
  83 |
  84 |     # ANTLR deps
--------------------
ERROR: failed to solve: process "/bin/sh -c bash /install/ubuntu_install_nnpack.sh" did not complete successfully: exit code: 2
ERROR: docker build failed.

Environment

Ubuntu 22.04
Latest TVM code at 1f4c568

Steps to reproduce

./docker/build.sh tvm.ci_cpu


### Triage

Please refer to the list of label tags [here](https://github.com/apache/tvm/wiki/Issue-Triage-Labels) to find the relevant tags and add them below in a bullet format (example below).

* needs-triage
@cissoidx cissoidx added needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it type: bug labels Jun 7, 2024
@cissoidx cissoidx changed the title [Bug] [Bug] ci_cpu docker image build fail Jun 7, 2024
@cissoidx
Copy link
Author

cissoidx commented Jun 7, 2024

the demo_cpu also fails

Building container (tvm.demo_cpu)...
[+] Building 1.7s (2/2) FINISHED                                                                                      docker:default
 => [internal] load build definition from Dockerfile.demo_cpu                                                                   0.0s
 => => transferring dockerfile: 1.37kB                                                                                          0.0s
 => ERROR [internal] load metadata for docker.io/tlcpack/ci-cpu:v0.55                                                           1.7s
------
 > [internal] load metadata for docker.io/tlcpack/ci-cpu:v0.55:
------
Dockerfile.demo_cpu:20
--------------------
  18 |     # Minimum docker image for demo purposes
  19 |     # prebuilt-image: tvmai/demo-cpu
  20 | >>> FROM tlcpack/ci-cpu:v0.55
  21 |
  22 |     COPY utils/apt-install-and-clear.sh /usr/local/bin/apt-install-and-clear
--------------------
ERROR: failed to solve: tlcpack/ci-cpu:v0.55: docker.io/tlcpack/ci-cpu:v0.55: not found
ERROR: docker build failed.

@yongwww
Copy link
Member

yongwww commented Jun 7, 2024

Thanks for reporting this error. Seems the image build failed with the same error for several days. cc: @lhutton1 @tqchen

@lhutton1
Copy link
Contributor

lhutton1 commented Jun 7, 2024

Looks related to this issue: Maratyszcza/NNPACK#219

@yongwww
Copy link
Member

yongwww commented Jun 8, 2024

if we don't have a quick fix for this, probably we could consider disabling nnpack.

@agoston-mc
Copy link
Contributor

During my testing, I could do a workaround by updating the ubuntu_install_nnpack.sh file (pastebin) so that it uses a previous commit of FP16 during the NNPACK cmake as a temporary solution, while they work it out in NNPACK.

@tqchen
Copy link
Member

tqchen commented Jun 12, 2024

i think it makes sense to disable and deprecate nnpack given it is more stale,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it type: bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants