diff --git a/README.md b/README.md index 83ea7e560d..6a7ba76d5d 100644 --- a/README.md +++ b/README.md @@ -284,6 +284,7 @@ Please read [getting_started](docs/en/get_started.md) for the basic usage of MMD - [Build for Win10](docs/en/01-how-to-build/windows.md) - [Build for Android](docs/en/01-how-to-build/android.md) - [Build for Jetson](docs/en/01-how-to-build/jetsons.md) + - [BUild for Jetson Docker](docs/en/01-how-to-build/Jetson_docker.md) - [Build for SNPE](docs/en/01-how-to-build/snpe.md) - [Cross Build for aarch64](docs/en/01-how-to-build/cross_build_ncnn_aarch64.md) - User Guide diff --git a/README_zh-CN.md b/README_zh-CN.md index e40d75200a..f6e9aadb68 100644 --- a/README_zh-CN.md +++ b/README_zh-CN.md @@ -268,6 +268,7 @@ MMDeploy 是 [OpenMMLab](https://openmmlab.com/) 模型部署工具箱,**为 - [Build for Win10](docs/zh_cn/01-how-to-build/windows.md) - [Build for Android](docs/zh_cn/01-how-to-build/android.md) - [Build for Jetson](docs/zh_cn/01-how-to-build/jetsons.md) + - [BUild for Jetson Docker](docs/zh_cn/01-how-to-build/Jetson_docker.md) - [Build for SNPE](docs/zh_cn/01-how-to-build/snpe.md) - [Cross Build for aarch64](docs/zh_cn/01-how-to-build/cross_build_ncnn_aarch64.md) - 使用 diff --git a/docker/Jetson/Jetpack4.6/Dockerfile b/docker/Jetson/Jetpack4.6/Dockerfile new file mode 100644 index 0000000000..115c095436 --- /dev/null +++ b/docker/Jetson/Jetpack4.6/Dockerfile @@ -0,0 +1,119 @@ +FROM nvcr.io/nvidia/l4t-pytorch:r32.7.1-pth1.10-py3 + +ARG MMDEPLOY_VERSION=main +ENV NVIDIA_VISIBLE_DEVICE all +ENV NVIDIA_DRIVER_CAPABILITIES all +ENV CUDA_HOME="/usr/local/cuda" +ENV PATH="/usr/local/cuda/bin:${PATH}" +ENV LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/lib/python3.8/dist-packages/opencv-python.libs/${LD_LIBRARY_PATH}" +ENV TENSORRT_DIR="/usr/include/aarch64-linux-gnu" + +ENV DEBIAN_FRONTEND=nointeractive +ENV FORCE_CUDA="1" + +USER root +WORKDIR /root/workspace + +# install dependencies && reinstall python3.8 +RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 42D5A192B819C5DA &&\ + apt-get remove python3 &&\ + apt-get update &&\ + apt-get install -y vim wget libspdlog-dev libssl-dev libpng-dev pkg-config libhdf5-100 libhdf5-dev patch --no-install-recommends\ + python3.8 python3.8-dev python3.8-pip --no-install-recommends &&\ + python3.8 -m pip install --upgrade --no-cache-dir setuptools packaging 'Cython<3' wheel &&\ + python3.8 -m pip install --no-cache-dir --verbose wget psutil numpy &&\ + python3.8 -m pip install --upgrade --force-reinstall --no-cache-dir --verbose cmake protobuf + python3.8 -m pip install onnx==1.10 versioned-hdf5 numpy + +# build pytorch 1.10.0 for python3.8 +# Hope it can works lol +# patch for https://github.com/pytorch/pytorch/issues/45323. Save here, maybe I will meet this issue. +# RUN PYTHON_ROOT=`pip3 show torch | grep Location: | cut -d' ' -f2` && \ +# TORCH_CMAKE_CONFIG=$PYTHON_ROOT/torch/share/cmake/Torch/TorchConfig.cmake && \ +# echo "patching _GLIBCXX_USE_CXX11_ABI in ${TORCH_CMAKE_CONFIG}" && \ +# sed -i 's/ set(TORCH_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=")/ set(TORCH_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=0")/g' ${TORCH_CMAKE_CONFIG} +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + libopenblas-dev \ + libopenmpi-dev \ + openmpi-bin \ + openmpi-common \ + gfortran \ + libomp-dev \ + && rm -rf /var/lib/apt/lists/* \ + && apt-get clean + +RUN git clone --branch v1.10.0 --depth=1 --recursive https://github.com/pytorch/pytorch /tmp/pytorch && \ + cd /tmp/pytorch && \ + wget https://gist.githubusercontent.com/dusty-nv/ce51796085178e1f38e3c6a1663a93a1/raw/4f1a0f948150c91f877aa38075835df748c81fe5/pytorch-1.10-jetpack-4.5.1.patch &&\ + patch -p1 < pytorch-1.10-jetpack-4.5.1.patch &&\ + export USE_NCCL=0 && \ + export USE_QNNPACK=0 && \ + export USE_PYTORCH_QNNPACK=0 && \ + export USE_NATIVE_ARCH=1 && \ + export USE_DISTRIBUTED=1 && \ + export USE_TENSORRT=0 && \ + python3.8 -m pip install -r requirements.txt && \ + python3.8 -m pip install --no-cache-dir scikit-build ninja && \ + python3.8 setup.py bdist_wheel && \ + cp dist/*.whl /root/workspace && \ + rm -rf /tmp/pytorch +RUN python3 -m pip install --verbose /opt/torch*.whl + +# build torchvision for python3.8 +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + libjpeg-dev \ + zlib1g-dev \ + && rm -rf /var/lib/apt/lists/* \ + && apt-get clean +RUN git clone --branch v0.11.1 --recursive --depth=1 https://github.com/pytorch/vision torchvision && \ + cd torchvision && \ + git checkout v0.11.1 && \ + python3.8 setup.py bdist_wheel && \ + cp dist/torchvision*.whl /opt && \ + rm -rf ../torchvision +RUN python3.8 -m pip install --no-cache-dir --verbose /opt/torchvision*.whl + +# build onnxruntime for python3.8 +RUN wget https://nvidia.box.com/shared/static/m9bz827ljmn771kvkjksdchmkczt3xke.whl -O onnxruntime_gpu-1.10.0-cp38-cp38-linux_aarch64.whl &&\ + python3.8 -m pip install --no-cache-dir onnxruntime_gpu-1.10.0-cp38-cp38-linux_aarch64.whl + +# install mmcv +RUN git clone --branch 2.x https://github.com/open-mmlab/mmcv.git +RUN cd mmcv &&\ + python3.8 -m pip install --no-cache-dir opencv-python==4.5.4.60 &&\ + MMCV_WITH_OPS=1 python3 -m pip install -e . + +# build ppl.cv +RUN git clone https://github.com/openppl-public/ppl.cv.git &&\ + echo "export PPLCV_DIR=/root/workspace/ppl.cv" >> ~/.bashrc &&\ + cd ppl.cv &&\ + ./build.sh cuda + +# build mmdeploy +RUN git clone --recursive -b $MMDEPLOY_VERSION --depth 1 https://github.com/open-mmlab/mmdeploy &&\ + cd mmdeploy &&\ + mkdir -p build && cd build &&\ + cmake .. \ + -DMMDEPLOY_TARGET_BACKENDS="trt" \ + -DTENSORRT_DIR=TENSORRT_DIR &&\ + make -j$(nproc) && make install && cd .. &&\ + python3 -m pip install --upgrade setuptools &&\ + python3 -m pip install -e . &&\ + mkdir -p build && cd build &&\ + cmake .. \ + -DMMDEPLOY_BUILD_SDK=ON \ + -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \ + -DMMDEPLOY_BUILD_EXAMPLES=ON \ + -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \ + -DMMDEPLOY_TARGET_BACKENDS="trt" \ + -DTENSORRT_DIR=TENSORRT_DIR \ + -Dpplcv_DIR=/root/workspace/ppl.cv/cuda-build/install/lib/cmake/ppl \ + -DMMDEPLOY_CODEBASES=all && \ + make -j$(nproc) && make install + +ENV MMDeploy_DIR="/root/workspace/mmdeploy/build/install/lib/cmake/MMDeploy" +ENV LD_LIBRARY_PATH="/root/workspace/mmdeploy/build/lib:${BACKUP_LD_LIBRARY_PATH}" +ENV PATH="/root/workspace/mmdeploy/build/bin:${PATH}" +ENV PYTHONPATH="/root/workspace/mmdeploy:${PYTHONPATH}" diff --git a/docker/Jetson/Jetpack5/Dockerfile b/docker/Jetson/Jetpack5/Dockerfile new file mode 100644 index 0000000000..a0f982d077 --- /dev/null +++ b/docker/Jetson/Jetpack5/Dockerfile @@ -0,0 +1,78 @@ +FROM nvcr.io/nvidia/l4t-pytorch:r35.2.1-pth2.0-py3 + +ARG MMDEPLOY_VERSION=main +ENV NVIDIA_VISIBLE_DEVICE all +ENV NVIDIA_DRIVER_CAPABILITIES all +ENV CUDA_HOME="/usr/local/cuda" +ENV PATH="/usr/local/cuda/bin:${PATH}" +ENV LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/lib/python3.8/dist-packages/opencv-python.libs${LD_LIBRARY_PATH}" +ENV TENSORRT_DIR="/usr/include/aarch64-linux-gnu" + +ENV DEBIAN_FRONTEND=nointeractive +ENV FORCE_CUDA="1" + +USER root +WORKDIR /root/workspace + +# install dependencies +RUN apt-get update &&\ + apt-get install -y vim wget libspdlog-dev libssl-dev libpng-dev pkg-config libhdf5-103 libhdf5-dev --no-install-recommends &&\ + python3 -m pip install onnx versioned-hdf5 + +# install onnxruntime +RUN wget https://nvidia.box.com/shared/static/mvdcltm9ewdy2d5nurkiqorofz1s53ww.whl -O onnxruntime_gpu-1.15.1.whl &&\ + python3 -m pip install --no-cache-dir onnxruntime_gpu-1.15.1-cp38-cp38-linux_aarch64.whl + +# install mmcv +RUN git clone --branch 2.x https://github.com/open-mmlab/mmcv.git &&\ + python3 -m pip install --no-cache-dir opencv-python==4.5.4.60 opencv-contrib-python==4.5.4.60 opencv-python-headless==4.5.4.60 &&\ + MMCV_WITH_OPS=1 python3 -m pip install -e . + +# build ppl.cv +RUN git clone https://github.com/openppl-public/ppl.cv.git &&\ + echo "export PPLCV_DIR=/root/workspace/ppl.cv" >> ~/.bashrc &&\ + ./build.sh cuda + +# build mmdeploy +RUN git clone --recursive -b $MMDEPLOY_VERSION --depth 1 https://github.com/open-mmlab/mmdeploy &&\ + cd mmdeploy &&\ + mkdir -p build && cd build &&\ + cmake .. \ + -DMMDEPLOY_TARGET_BACKENDS="trt" \ + -DTENSORRT_DIR=TENSORRT_DIR &&\ + make -j$(nproc) && make install && cd .. &&\ + cd mmdeploy &&\ + python3 -m pip install --upgrade setuptools &&\ + python3 -m pip install -e . &&\ + mkdir -p build && cd build &&\ + cmake .. \ + -DMMDEPLOY_BUILD_SDK=ON \ + -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \ + -DMMDEPLOY_BUILD_EXAMPLES=ON \ + -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \ + -DMMDEPLOY_TARGET_BACKENDS="trt" \ + -DTENSORRT_DIR=TENSORRT_DIR \ + -Dpplcv_DIR=/root/workspace/ppl.cv/cuda-build/install/lib/cmake/ppl \ + -DMMDEPLOY_CODEBASES=all && \ + make -j$(nproc) && make install + +# add patch to solve the build error +RUN sed -i '/def _run_symbolic_method(g, op_name, symbolic_fn, args):/,/except TypeError as e:/\ + { + s/\ + return symbolic_fn(g, \*args)/\ + graph_context = jit_utils.GraphContext(\ + graph=g,\ + block=g.block(),\ + opset=GLOBALS.export_onnx_opset_version,\ + original_node=None, # type: ignore[arg-type]\ + params_dict=_params_dict,\ + env={},\ + )\ + return symbolic_fn(graph_context, \*args)/\ + }' /usr/local/lib/python3.8/dist-packages/torch/onnx/symbolic_helper.py + +ENV MMDeploy_DIR="/root/workspace/mmdeploy/build/install/lib/cmake/MMDeploy" +ENV LD_LIBRARY_PATH="/root/workspace/mmdeploy/build/lib:${BACKUP_LD_LIBRARY_PATH}" +ENV PATH="/root/workspace/mmdeploy/build/bin:${PATH}" +ENV PYTHONPATH="/root/workspace/mmdeploy:${PYTHONPATH}" diff --git a/docs/en/01-how-to-build/Jetson_docker.md b/docs/en/01-how-to-build/Jetson_docker.md new file mode 100644 index 0000000000..c583c88b7f --- /dev/null +++ b/docs/en/01-how-to-build/Jetson_docker.md @@ -0,0 +1,134 @@ +# Use Jetson Docker Image + +This document guides how to install mmdeploy with [Docker](https://docs.docker.com/get-docker/) on Jetson. + +## Get prebuilt docker images + +MMDeploy provides prebuilt docker images for the convenience of its users on [Docker Hub](https://hub.docker.com/r/openmmlab/mmdeploy). The docker images are built on +the latest and released versions. We release two docker version, for Jetpack=5.1 and Jetpack=4.6.1 +For instance, the image with tag `openmmlab/mmdeploy_jetpack5:v1` is built for Jetpack5.1 and the image with tag `openmmlab/mmdeploy_jetpack4.6.1:v1` is build for Jetpack 4.6.1. +The specifications of the Docker Images are shown below. + +- jetpack5.1 + +| Item | Version | +| :---------: | :---------: | +| Jetpack | 5.1 | +| Python | 3.8.10 | +| Torch | 2.0.0 | +| TorchVision | 0.15.0 | + +- jetpack4.6.1 + +| Item | Version | +| :---------: | :---------: | +| Jetpack | 4.6.1 | +| Python | 3.8.10 | +| Torch | 1.10.0 | +| TorchVision | 0.11.0 | + +- jetpack 5.1 +```shell +export TAG=openmmlab/mmdeploy_jetpack5:v1 +docker pull $TAG +``` +- jetpack 4.6.1 +```shell +export TAG=openmmlab/mmdeploy_jetpack4.6:v1 +docker pull $TAG +``` +## Build docker images (optional) + +If the prebuilt docker images do not meet your requirements, +then you can build your own image by running the following script. +The docker file is `docker/jetson/jetpack5/Dockerfile` and `docker/jetson/jetpack4.6/Dockerfile`, + +```shell +docker build docker/jetson/jetpack5 -t openmmlab/mmdeploy_jetpack5:v1 . +// +docker build docker/jetson/jetpack4.6 -t openmmlab/mmdeploy_jetpack4.6:v1 . +``` + +## Run docker container + +After pulling or building the docker image, you can use `docker run` to launch the docker service: + +```shell +docker run -it --rm --runtime nvidia --network host openmmlab/mmdeploy_jetpack5:v1 +// +docker run -it --rm --runtime nvidia --network host openmmlab/mmdeploy_jetpack4.6:v1 +``` + +## TroubleShooting +update: I solved the problem 3, 4 using sed in docker. +If you using the jetpack5, it has some question need to solve. +1. OpenCV problem + if you find import cv2 wrong, can't find the libpng15.so +```shell + ln -s /usr/local/lib/python3.x/dist-packages/opencv-python.libs/* /usr/lib +``` + +2. mmdetection problem + if you find installed the mmdetection, but import the mmdet failed. you should use this to install +```shell + python3 -m pip install --user -e . +``` + +3. Jetson No distributed problem(this is rewrited with the PR) + if you convert the model like [Jetson.md](https://github.com/open-mmlab/mmdeploy/blob/main/docs/en/01-how-to-build/jetsons.md) + you may find torch.distributed has no attribute ReduceOp. + I just issue and make a simple patch, add file jetson_patch.py on ./mmdeploy/tools/ +```python +import torch.distributed +if not torch.distributed.is_available(): + torch.distributed.ReduceOp = lambda: None +``` + and import jetson_patch at the beginning which file you want. + I know is not quietly ellegant, but it works well...(for Jetson AGX Orin) + +4. Jetpack with PyTorch 2.0 has some issue +> If you use the docker, we help you change the PyTorch in dockerfile. + + we need to modify torch.onnx._run_symbolic_method + **from** +```python +def _run_symbolic_method(g, op_name, symbolic_fn, args): + r""" + This trampoline function gets invoked for every symbolic method + call from C++. + """ + try: + return symbolic_fn(g, *args) + except TypeError as e: + # Handle the specific case where we didn't successfully dispatch + # to symbolic_fn. Otherwise, the backtrace will have the clues + # you need. + e.args = ("{} (occurred when translating {})".format(e.args[0], op_name),) + raise +``` + **to** +```python +@_beartype.beartype +def _run_symbolic_method(g, op_name, symbolic_fn, args): + r""" + This trampoline function gets invoked for every symbolic method + call from C++. + """ + try: + graph_context = jit_utils.GraphContext( + graph=g, + block=g.block(), + opset=GLOBALS.export_onnx_opset_version, + original_node=None, # type: ignore[arg-type] + params_dict=_params_dict, + env={}, + ) + return symbolic_fn(graph_context, *args) + except TypeError as e: + # Handle the specific case where we didn't successfully dispatch + # to symbolic_fn. Otherwise, the backtrace will have the clues + # you need. + e.args = (f"{e.args[0]} (occurred when translating {op_name})",) + raise + ``` +Finally we can use Jetpack5.1 && MMDeploy happily:) diff --git a/docs/zh_cn/01-how-to-build/Jetson_docker.md b/docs/zh_cn/01-how-to-build/Jetson_docker.md new file mode 100644 index 0000000000..233334a063 --- /dev/null +++ b/docs/zh_cn/01-how-to-build/Jetson_docker.md @@ -0,0 +1,123 @@ +# 使用Jetson Docker镜像 + +本文档指导如何在Jetson上通过[Docker](https://docs.docker.com/get-docker/)安装mmdeploy。 + +## 获取预构建的docker镜像 + +MMDeploy为用户在[Docker Hub](https://hub.docker.com/r/openmmlab/mmdeploy)上提供了预构建的docker镜像。这些docker镜像基于最新和已发布的版本构建。我们发布了两个版本的docker镜像,分别为Jetpack=5.1和Jetpack=4.6.1。例如,标签为`openmmlab/mmdeploy_jetpack5:v1`的镜像是为Jetpack5.1构建的,而标签为`openmmlab/mmdeploy_jetpack4.6.1:v1`的镜像则是为Jetpack 4.6.1构建的。Docker镜像的规格如下所示。 + +- jetpack5.1 + +| 项目 | 版本 | +| :--------: | :------: | +| Jetpack | 5.1 | +| Python | 3.8.10 | +| Torch | 2.0.0 | +| TorchVision| 0.15.0 | + +- jetpack4.6.1 + +| 项目 | 版本 | +| :--------: | :------: | +| Jetpack | 4.6.1 | +| Python | 3.8.10 | +| Torch | 1.10.0 | +| TorchVision| 0.11.0 | + +- jetpack 5.1 +```shell +export TAG=openmmlab/mmdeploy_jetpack5:v1 +docker pull $TAG +``` +- jetpack 4.6.1 +```shell +export TAG=openmmlab/mmdeploy_jetpack4.6:v1 +docker pull $TAG +``` + +## 构建docker镜像(可选) +如果预构建的docker镜像不符合您的要求,您可以通过运行以下脚本来构建自己的镜像。docker文件分别为docker/jetson/jetpack5/Dockerfile和docker/jetson/jetpack4.6/Dockerfile, + +```shell +docker build docker/jetson/jetpack5 -t openmmlab/mmdeploy_jetpack5:v1 . +// +docker build docker/jetson/jetpack4.6 -t openmmlab/mmdeploy_jetpack4.6:v1 . +``` + +## 运行docker容器 +拉取或构建docker镜像后,您可以使用docker run来启动docker服务: + +```shell +docker run -it --rm --runtime nvidia --network host openmmlab/mmdeploy_jetpack5:v1 +// +docker run -it --rm --runtime nvidia --network host openmmlab/mmdeploy_jetpack4.6:v1 +``` + +## 故障排除 +如果您使用的是jetpack5,可能需要解决一些问题。 + +1. OpenCV问题 +如果您发现import cv2时出错,找不到libpng15.so +```shell + ln -s /usr/local/lib/python3.x/dist-packages/opencv-python.libs/* /usr/lib +``` +2. mmdetection问题 +如果您发现安装了mmdetection,但无法导入mmdet。您可以使用以下命令来安装 +```shell + python3 -m pip install --user -e . +``` +3. Jetson分布式问题(已在PR中重写) +如果您按照Jetson.md中的方法转换模型,您可能会发现torch.distributed没有ReduceOp属性。我只是提出了问题并做了一个简单的补丁,在./mmdeploy/tools/下添加jetson_patch.py文件 +```python +import torch.distributed +if not torch.distributed.is_available(): + torch.distributed.ReduceOp = lambda: None +``` +并在您需要的文件开头导入jetson_patch。我知道这并不优雅,但它确实有效...(适用于Jetson AGX Orin) + +4. Jetpack5.1 PyTorch2.0有一些问题 +> 如果您直接使用我们的预编译docker镜像或者使用dockerfile进行构建镜像,这个问题已在dockerfile中解决 + + 我们需要修改 torch.onnx._run_symbolic_method 这个函数。 + **从** +```python +def _run_symbolic_method(g, op_name, symbolic_fn, args): + r""" + This trampoline function gets invoked for every symbolic method + call from C++. + """ + try: + return symbolic_fn(g, *args) + except TypeError as e: + # Handle the specific case where we didn't successfully dispatch + # to symbolic_fn. Otherwise, the backtrace will have the clues + # you need. + e.args = ("{} (occurred when translating {})".format(e.args[0], op_name),) + raise +``` + **到** +```python +@_beartype.beartype +def _run_symbolic_method(g, op_name, symbolic_fn, args): + r""" + This trampoline function gets invoked for every symbolic method + call from C++. + """ + try: + graph_context = jit_utils.GraphContext( + graph=g, + block=g.block(), + opset=GLOBALS.export_onnx_opset_version, + original_node=None, # type: ignore[arg-type] + params_dict=_params_dict, + env={}, + ) + return symbolic_fn(graph_context, *args) + except TypeError as e: + # Handle the specific case where we didn't successfully dispatch + # to symbolic_fn. Otherwise, the backtrace will have the clues + # you need. + e.args = (f"{e.args[0]} (occurred when translating {op_name})",) + raise + ``` +最后您就可以开心的使用这个镜像了:) diff --git a/mmdeploy/pytorch/functions/distribute.py b/mmdeploy/pytorch/functions/distribute.py new file mode 100644 index 0000000000..a069d175f7 --- /dev/null +++ b/mmdeploy/pytorch/functions/distribute.py @@ -0,0 +1,12 @@ +# Copyright (c) OpenMMLab. All rights reserved. +from typing import Optional +import torch + +from mmdeploy.core import FUNCTION_REWRITER + +@FUNCTION_REWRITER.register_rewriter(func_name='torch.distributed') +def distributed_rewriter(): + """rewrite torch.distributed to support some embedding device for higher PyTorch""" + # check torch.distributed is available? + if not torch.distributed.is_available(): + torch.distributed.ReduceOp = lambda: None