Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v2.4.10 build error #218

Open
Amphetaminewei opened this issue Oct 30, 2024 · 6 comments
Open

v2.4.10 build error #218

Amphetaminewei opened this issue Oct 30, 2024 · 6 comments
Assignees
@Presburger

Comments

@Amphetaminewei
Copy link

I got the following error when trying to build v2.4.10 locally:


-- Conan: Dependency gRPC already found
-- Conan: Dependency nlohmann_json already found
-- Conan: Dependency opentelemetry-proto already found
-- Conan: Dependency Protobuf already found
-- Conan: Dependency OpenSSL already found
-- Conan: Dependency Thrift already found
-- Conan: Dependency Boost already found
-- Conan: Dependency CURL already found
-- Conan: Dependency absl already found
-- Library opentelemetry_exporter_ostream_metrics found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_ostream_metrics.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_exporter_ostream_metrics_opentelemetry_exporter_ostream_metrics
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_ostream_metrics.a
-- Library opentelemetry_metrics found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_metrics.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_metrics_opentelemetry_metrics
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_metrics.a
-- Library opentelemetry_common found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_common.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_common_opentelemetry_common
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_common.a
-- Library opentelemetry_resources found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_resources.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_resources_opentelemetry_resources
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_resources.a
-- Library opentelemetry_http_client_curl found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_http_client_curl.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_http_client_curl_opentelemetry_http_client_curl
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_http_client_curl.a
-- Library opentelemetry_exporter_jaeger_trace found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_jaeger_trace.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_exporter_jaeger_trace_opentelemetry_exporter_jaeger_trace
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_jaeger_trace.a
-- Library opentelemetry_trace found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_trace.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_trace_opentelemetry_trace
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_trace.a
-- Library opentelemetry_exporter_zipkin_trace found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_zipkin_trace.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_exporter_zipkin_trace_opentelemetry_exporter_zipkin_trace
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_zipkin_trace.a
-- Library opentelemetry_proto found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_proto.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_proto_opentelemetry_proto
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_proto.a
-- Library opentelemetry_exporter_otlp_http_client found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_http_client.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_exporter_otlp_http_client_opentelemetry_exporter_otlp_http_client
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_http_client.a
-- Library opentelemetry_otlp_recordable found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_otlp_recordable.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_otlp_recordable_opentelemetry_otlp_recordable
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_otlp_recordable.a
-- Library opentelemetry_exporter_otlp_http_metric found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_http_metric.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_exporter_otlp_http_metric_opentelemetry_exporter_otlp_http_metric
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_http_metric.a
-- Library opentelemetry_exporter_otlp_http found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_http.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_exporter_otlp_http_opentelemetry_exporter_otlp_http
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_http.a
-- Library opentelemetry_exporter_otlp_grpc_client found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_grpc_client.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_exporter_otlp_grpc_client_opentelemetry_exporter_otlp_grpc_client
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_grpc_client.a
-- Library opentelemetry_exporter_otlp_grpc_metrics found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_grpc_metrics.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_exporter_otlp_grpc_metrics_opentelemetry_exporter_otlp_grpc_metrics
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_grpc_metrics.a
-- Library opentelemetry_exporter_otlp_grpc found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_grpc.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_exporter_otlp_grpc_opentelemetry_exporter_otlp_grpc
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_otlp_grpc.a
-- Library opentelemetry_version found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_version.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_version_opentelemetry_version
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_version.a
-- Library opentelemetry_exporter_ostream_span found /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_ostream_span.a
-- Skipping already existing target: CONAN_LIB::opentelemetry-cpp_opentelemetry_exporter_ostream_span_opentelemetry_exporter_ostream_span
-- Found: /home/wangweinan/.conan/data/opentelemetry-cpp/1.8.1.1/milvus/dev/package/7f893476508dc11d0dadc4d7f6c8952b97edd9cc/lib/libopentelemetry_exporter_ostream_span.a
-- knowhere src compile options: 
-- Adding -Og to compile flag
-- Configuring incomplete, errors occurred!
Traceback (most recent call last):
  File "/home/wangweinan/.local/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/home/wangweinan/.local/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/wangweinan/.local/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/build_meta.py", line 434, in build_wheel
    return _build(['bdist_wheel'])
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/build_meta.py", line 425, in _build
    return self._build_with_temp_dir(
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/build_meta.py", line 406, in _build_with_temp_dir
    self.run_setup()
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/build_meta.py", line 521, in run_setup
    super().run_setup(setup_script=setup_script)
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/build_meta.py", line 319, in run_setup
    exec(code, locals())
  File "<string>", line 149, in <module>
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 183, in setup
    return run_commands(dist)
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
    dist.run_commands()
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/dist.py", line 999, in run_command
    super().run_command(command)
  File "/tmp/build-env-pa9z5m79/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "<string>", line 112, in run
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/wangweinan/build/milvus-lite', '-DENABLE_UNIT_TESTS=OFF']' returned non-zero exit status 1.

ERROR Backend subprocess exited when trying to invoke build_wheel

Here's how I built it:

git clone --recurse-submodules https://github.com/milvus-io/milvus-lite.git
cd milvus-lite/python
git checkout v2.4.10
python3 -m build --wheel

Is it what compilation parameters I'm missing?
@Amphetaminewei
Copy link
Author

We are trying to build the deb package for Milvus lite, this thing is not easy for us, because we have not used conan, do you have any experience in this area?

@Amphetaminewei
Copy link
Author

I also made some changes during the build process, my BLAS and LAPACK versions are relatively low(3.9.0), only the .pc file does not have a cmake file, and it will report an error when building knowhere, so I made the following changes:

wangweinan@wangweinan-pc:~/prj/milvus/milvus-lite/build$ git show
commit 6311fdb8a310e5c190182d84e89a716b7e05977a (HEAD -> test-build)
Author: Wei Wei <[email protected]>
Date:   Wed Oct 30 20:30:01 2024 +0800

    feat: add knowhere patch

diff --git a/thirdparty/knowhere.patch b/thirdparty/knowhere.patch
new file mode 100644
index 0000000..738f1ca
--- /dev/null
+++ b/thirdparty/knowhere.patch
@@ -0,0 +1,16 @@
+cmake/libs/libfaiss.cmake
+--- a/cmake/libs/libfaiss.cmake        2024-10-30 20:18:09.276957831 +0800
++++ b/cmake/libs/libfaiss.cmake        2024-10-30 20:19:27.346054014 +0800
+@@ -67,9 +67,9 @@
+   set(BLA_VENDOR Apple)
+ endif()
+
+-find_package(BLAS REQUIRED)
+-
+-find_package(LAPACK REQUIRED)
++find_package(PkgConfig REQUIRED)
++pkg_check_modules(BLAS REQUIRED blas-netlib)
++pkg_check_modules(LAPACK REQUIRED lapack-netlib)
+
+ if(__X86_64)
+   list(REMOVE_ITEM FAISS_SRCS ${FAISS_AVX2_SRCS})
\ No newline at end of file
diff --git a/thirdparty/milvus-knowhere.patch b/thirdparty/milvus-knowhere.patch
new file mode 100644
index 0000000..b3f90ec
--- /dev/null
+++ b/thirdparty/milvus-knowhere.patch
@@ -0,0 +1,17 @@
+diff --git a/internal/core/thirdparty/knowhere/CMakeLists.txt b/internal/core/thirdparty/knowhere/CMakeLists.txt
+index 745842432c..02f0475b40 100644
+--- a/internal/core/thirdparty/knowhere/CMakeLists.txt
++++ b/internal/core/thirdparty/knowhere/CMakeLists.txt
+@@ -49,6 +49,12 @@ FetchContent_GetProperties( knowhere )
+ if ( NOT knowhere_POPULATED )
+     FetchContent_Populate( knowhere )
+ 
++    message( STATUS "Applying patch to knowhere" )
++    execute_process(
++    COMMAND patch -p1 -i ${CMAKE_SOURCE_DIR}/thirdparty/knowhere.patch
++    WORKING_DIRECTORY ${knowhere_SOURCE_DIR}
++    )
++
+     # Adding the following target:
+     # knowhere
+     add_subdirectory( ${knowhere_SOURCE_DIR}

Is the knowhere build error related to my BLAS and LAPACK versions? How can I view the more detailed compilation logs?

@Amphetaminewei
Copy link
Author

I'm trying to compile knowhere separately, and I found that there is a network-related error when performing conan install when compiling alone, maybe the error of compiling Milvus lite is also caused by a network issue

@Amphetaminewei
Copy link
Author

The problem may not be related to knowhere, I found more errors in the logs:

CMake Error at build/Finds2n.cmake:181 (include):
  include could not find requested file:

    /home/wangweinan/.conan/data/s2n/1.3.15/_/_/package/dfe12eae9cbeecab0fd96fa1b6854ee27b02fb2b/lib/cmake/conan-official-s2n-targets.cmake
Call Stack (most recent call first):
  /home/wangweinan/cmake-3.26.5-linux-x86_64/share/cmake-3.26/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
  build/Findaws-c-io.cmake:186 (find_dependency)
  /home/wangweinan/cmake-3.26.5-linux-x86_64/share/cmake-3.26/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
  build/Findaws-c-http.cmake:186 (find_dependency)
  /home/wangweinan/cmake-3.26.5-linux-x86_64/share/cmake-3.26/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
  build/Findaws-crt-cpp.cmake:172 (find_dependency)
  /home/wangweinan/cmake-3.26.5-linux-x86_64/share/cmake-3.26/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
  build/FindAWSSDK.cmake:780 (find_dependency)
  /home/wangweinan/cmake-3.26.5-linux-x86_64/share/cmake-3.26/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
  build/FindArrow.cmake:252 (find_dependency)
  CMakeLists.txt:105 (find_package)

Looks like it's an issue with my conan environment

@Presburger
Copy link
Collaborator

@Amphetaminewei Thank you for your interest in the Milvus-lite project. When compiling Milvus-lite, I have provided a Manylinux Docker image and recommend compiling under manylinux. The wheel package compiled with manylinux offers good compatibility and can run on most newer Linux platforms.

@Amphetaminewei
Copy link
Author

Hi @Presburger , thank you for your reply, where can I get the link to the Docker image? Have you ever tried to make a deb package out of Milvus lite? Similar to what the Ubuntu community does, dependencies are managed through deb packages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Presburger Presburger

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Amphetaminewei @Presburger