-
Notifications
You must be signed in to change notification settings - Fork 508
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request] Code and build system for multiple backends #3120
Comments
See #3120. - CMake: add `ENABLE_TENSORFLOW` and `ENABLE_PYTORCH`. `BUILD_TENSORFLOW` will be enabled when `TENSORFLOW_ROOT` is not empty or `USE_TF_PYTHON_LIBS` is on. - api_cc: add `BUILD_TENSORFLOW` and `BUILD_PYTORCH` defination. Move several functions from `common.h` to `commonTF.h` to prevent exposing them to header files. - CI: download libtorch in the build/test CC actions. --------- Signed-off-by: Jinzhe Zeng <[email protected]>
* Fix max nbor size related issues (deepmodeling#3157) * Merge master into devel (deepmodeling#3167) * [pre-commit.ci] pre-commit autoupdate (deepmodeling#3163) <!--pre-commit.ci start--> updates: - [github.com/astral-sh/ruff-pre-commit: v0.1.13 → v0.1.14](astral-sh/ruff-pre-commit@v0.1.13...v0.1.14) <!--pre-commit.ci end--> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * setup PyTorch C++ interface build environement (deepmodeling#3169) See deepmodeling#3120. - CMake: add `ENABLE_TENSORFLOW` and `ENABLE_PYTORCH`. `BUILD_TENSORFLOW` will be enabled when `TENSORFLOW_ROOT` is not empty or `USE_TF_PYTHON_LIBS` is on. - api_cc: add `BUILD_TENSORFLOW` and `BUILD_PYTORCH` defination. Move several functions from `common.h` to `commonTF.h` to prevent exposing them to header files. - CI: download libtorch in the build/test CC actions. --------- Signed-off-by: Jinzhe Zeng <[email protected]> * docs: add TF icons to platform-specific features (deepmodeling#3171) Fix deepmodeling#3121. The PyTorch icon can be added when a feature implemented by PyTorch is added. However, I can't find a way to add an icon to TOC. ![image](https://github.com/deepmodeling/deepmd-kit/assets/9496702/7f29da27-af81-4850-9da0-79310d216b2d) Signed-off-by: Jinzhe Zeng <[email protected]> * add universal Python inference interface DeepPot (deepmodeling#3164) Need discussion for other classes. --------- Signed-off-by: Jinzhe Zeng <[email protected]> * detect version in advance before building deepmd-kit-cu11 (deepmodeling#3172) Fix deepmodeling#3168. See: pypa/setuptools-scm#1006 (comment) --------- Signed-off-by: Jinzhe Zeng <[email protected]> --------- Signed-off-by: Jinzhe Zeng <[email protected]> Co-authored-by: Denghui Lu <[email protected]> Co-authored-by: Jinzhe Zeng <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Fix deepmodeling#3120. Signed-off-by: Jinzhe Zeng <[email protected]>
After investigation, the pre-built package seems impossible due to: (1) PyTorch from the PyPI has incompatible
|
Fix #3120. One can disable building the TensorFlow backend during `pip install` by setting `DP_ENABLE_TENSORFLOW=0`. --------- Signed-off-by: Jinzhe Zeng <[email protected]>
* throw errors when PyTorch CXX11 ABI is different from TensorFlow (deepmodeling#3201) If so, throw the following error: ``` -- PyTorch CXX11 ABI: 0 CMake Error at CMakeLists.txt:162 (message): PyTorch CXX11 ABI mismatch TensorFlow: 0 != 1 ``` Signed-off-by: Jinzhe Zeng <[email protected]> * allow disabling TensorFlow backend during Python installation (deepmodeling#3200) Fix deepmodeling#3120. One can disable building the TensorFlow backend during `pip install` by setting `DP_ENABLE_TENSORFLOW=0`. --------- Signed-off-by: Jinzhe Zeng <[email protected]> * breaking: pt: add dp model format and refactor pt impl for the fitting net. (deepmodeling#3199) - add dp model format (backend independent definition) for the fitting - refactor torch support, compatible with dp model format - fix mlp issue: the idt should only be used when a skip connection is available. - add tools `to_numpy_array` and `to_torch_tensor`. --------- Co-authored-by: Han Wang <[email protected]> * remove duplicated fitting output check. fix codeql (deepmodeling#3202) Co-authored-by: Han Wang <[email protected]> --------- Signed-off-by: Jinzhe Zeng <[email protected]> Co-authored-by: Jinzhe Zeng <[email protected]> Co-authored-by: Han Wang <[email protected]> Co-authored-by: Han Wang <[email protected]>
Summary
This proposal describes how multiple backends are organized and built.
Detailed Description
Code structure
The code may organized as
(updated: Jan 27, 2024)
deepmd
is actuallydeepmd_tf
, but we must keep compatibility.Other codes are not affected.
CMake
Currently, CMake has
TENSORFLOW_ROOT
andUSE_TF_PYTHON_LIBS
. Now they are not required if one enablesUSE_PT_PYTHON_LIBS
. However, an error should be raised if one does not enable any backend.One or two backends can be built, as shown below:
Pip
By default, both backends are installed. This doesn't bring great changes because PT codes don't use C++.
Add a new environment variable option to disable TF.
Add optional dependencies for PyTorch, i.e.
Pre-built packages with dependencies
As both TensorFlow and PyTorch are pretty large in size, we need to determine whether to bring them into the same pre-built package...
Further Information, Files, and Links
No response
The text was updated successfully, but these errors were encountered: