Skip to content

Commit

Permalink
[CI][Update] Update CI Actions for releasing a brand new release pack…
Browse files Browse the repository at this point in the history
…age. (#1219)

* Add compilation script for protobuf
* Add Docker compilation script for paddle2onnx
* Upgrade version is good
* Support automatic building and uploading of Python packages during release version publication
  • Loading branch information
Zheng-Bicheng authored Apr 13, 2024
1 parent b32b7ba commit 83ecf73
Show file tree
Hide file tree
Showing 10 changed files with 155 additions and 27 deletions.
File renamed without changes.
42 changes: 42 additions & 0 deletions .github/workflows/release_linux_x86_64.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: Build Package

on:
release:
types: [published]

permissions:
contents: read

jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [ '3.8', '3.9', '3.10', '3.11', '3.12']
architecture: [ 'x64' ]

# For the sake of simplicity in testing, the Paddle2ONNX packaging program will temporarily only package executable files for Python 3.8.
# In the future, we will need to extend support to cover Python 3.8 through Python 3.10.
steps:
# Checkout the latest branch of Paddle2ONNX.
- name: Checkout Paddle2ONNX
uses: actions/checkout@v4
with:
submodules: true

- name: Build on manylinux2014_x86_64
uses: docker://quay.io/pypa/manylinux2014_x86_64:latest
with:
entrypoint: bash
args: .github/workflows/scripts/entrypoint.sh ${{ matrix.python-version }} manylinux2014_x86_64

- uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
with:
name: wheels
path: dist

- name: Publish package
uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
with:
user: __token__
password: ${{ secrets.ZHENG_BICHENG_PYPI_TOKEN }}
22 changes: 22 additions & 0 deletions .github/workflows/scripts/build_protobuf_unix.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/bin/bash

# Copyright (c) ONNX Project Contributors
#
# SPDX-License-Identifier: Apache-2.0

export CORE_NUMBER=$1

if [[ -z "$CORE_NUMBER" ]]; then
export CORE_NUMBER=1
fi

# Build protobuf from source with -fPIC on Unix-like system
ORIGINAL_PATH=$(pwd)
cd ..
git clone https://github.com/protocolbuffers/protobuf.git -b v3.16.0
cd protobuf
mkdir build_source && cd build_source
cmake ../cmake -DCMAKE_INSTALL_PREFIX=${PWD}/installed_protobuf_lib -Dprotobuf_BUILD_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -Dprotobuf_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE=Release
make -j$CORE_NUMBER && make install
export PATH=${PWD}/installed_protobuf_lib/bin:${PATH}
cd $ORIGINAL_PATH
54 changes: 54 additions & 0 deletions .github/workflows/scripts/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/bin/bash

# Copyright (c) ONNX Project Contributors
#
# SPDX-License-Identifier: Apache-2.0

set -e -x

# CLI arguments
PY_VERSION=$1
PLAT=$2
GITHUB_EVENT_NAME=$3

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib

# Compile wheels
# Need to be updated if there is a new Python Version
if [ "$(uname -m)" == "aarch64" ]; then
PIP_INSTALL_COMMAND="$PY_VERSION -m pip install --no-cache-dir -q"
PYTHON_COMMAND="$PY_VERSION"
else
declare -A python_map=( ["3.8"]="cp38-cp38" ["3.9"]="cp39-cp39" ["3.10"]="cp310-cp310" ["3.11"]="cp311-cp311" ["3.12"]="cp312-cp312")
PY_VER=${python_map[$PY_VERSION]}
PIP_INSTALL_COMMAND="/opt/python/${PY_VER}/bin/pip install --no-cache-dir -q"
PYTHON_COMMAND="/opt/python/${PY_VER}/bin/python"
fi

# Update pip and install cmake
$PIP_INSTALL_COMMAND --upgrade pip
$PIP_INSTALL_COMMAND cmake

# Build protobuf from source
source .github/workflows/scripts/build_protobuf_unix.sh "$(nproc)"

# Build Paddle2ONNX wheels
$PYTHON_COMMAND -m build --wheel || { echo "Building wheels failed."; exit 1; }

# Bundle external shared libraries into the wheels
# find -exec does not preserve failed exit codes, so use an output file for failures
failed_wheels=$PWD/failed-wheels
rm -f "$failed_wheels"
find . -type f -iname "*-linux*.whl" -exec sh -c "auditwheel repair '{}' -w \$(dirname '{}') --plat '${PLAT}' || { echo 'Repairing wheels failed.'; auditwheel show '{}' >> '$failed_wheels'; }" \;

if [[ -f "$failed_wheels" ]]; then
echo "Repairing wheels failed:"
cat failed-wheels
exit 1
fi

# Remove useless *-linux*.whl; only keep manylinux*.whl
rm -f dist/*-linux*.whl

echo "Successfully build wheels:"
find . -type f -iname "*manylinux*.whl"
4 changes: 1 addition & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,11 @@ TensorRT/OpenVINO/MNN/TNN/NCNN,以及其它对 ONNX 开源格式进行支持

# 3 安装

针对PaddlePaddle2.5.2的用户可以直接运行以下命令行代码来安装P2O

```
pip install paddle2onnx
```

由于没有自动发包机制,针对PaddlePaddle2.6.0的用户,请按照[Github 源码安装方式](docs/zh/compile.md)编译Paddle2ONNX。
开发用户,请按照[Github 源码安装方式](docs/zh/compile_local.md)编译Paddle2ONNX。

# 4 使用

Expand Down
2 changes: 1 addition & 1 deletion VERSION_NUMBER
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.1.0
1.2.0
19 changes: 0 additions & 19 deletions docs/zh/change_log.md

This file was deleted.

26 changes: 26 additions & 0 deletions docs/zh/compile_docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Docker编译安装Paddle2ONNX

Paddle2ONNX编译安装需要确保环境满足以下需求
- cmake >= 3.18.0
- protobuf == 3.16.0

注意:Paddle2ONNX产出的模型,在使用ONNX Runtime推理时,要求使用最新版本(1.10.0版本以及上),如若需要使用低版本(1.6~1.10之间),则需要将ONNX版本降至1.8.2,在执行完`git submodule update`后,执行如下命令,然后再进行编译
```
cd Paddle2ONNX/third/onnx
git checkout v1.8.1
```

拉取manylinux镜像并创建容器

```bash
docker pull quay.io/pypa/manylinux2014_x86_64
docker run --name p2o_build -d quay.io/pypa/manylinux2014_x86_64
```

创建容器并运行

```bash
docker start p2o_build
docker exec -it p2o_build bash
```

13 changes: 9 additions & 4 deletions docs/zh/compile.md → docs/zh/compile_local.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# 编译安装Paddle2ONNX
# 本地编译安装Paddle2ONNX

Paddle2ONNX编译安装需要确保环境满足以下需求
- cmake >= 3.18.0
- protobuf >= 3.16.0
- protobuf == 3.16.0

注意:Paddle2ONNX产出的模型,在使用ONNX Runtime推理时,要求使用最新版本(1.10.0版本以及上),如若需要使用低版本(1.6~1.10之间),则需要将ONNX版本降至1.8.2,在执行完`git submodule update`后,执行如下命令,然后再进行编译
```
Expand All @@ -13,6 +13,7 @@ git checkout v1.8.1
## Linux/Mac编译安装

### 安装Protobuf

```
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
Expand All @@ -25,26 +26,30 @@ make install
# 将编译目录加入环境变量
export PATH=${PWD}/installed_protobuf_lib/bin:${PATH}
```

### 安装Paddle2ONNX

```
git clone https://github.com/PaddlePaddle/Paddle2ONNX.git
cd Paddle2ONNX
git submodule init
git submodule update
python -m build
python -m pip install dist/*.whl
```

## Windows编译安装

注意Windows编译安装先验条件是系统中已安装好Visual Studio 2019

### 打开VS命令行工具

系统菜单中,找到**x64 Native Tools Command Prompt for VS 2019**打开

### 安装Protobuf

注意下面cmake命令中`-DCMAKE_INSTALL_PREFIX`指定为你实际设定的路径

```
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
Expand All @@ -59,12 +64,12 @@ set PATH=D:\Paddle\installed_protobuf_lib\bin;%PATH%
```

### 安装Paddle2ONNX

```
git clone https://github.com/PaddlePaddle/Paddle2ONNX.git
cd Paddle2ONNX
git submodule init
git submodule update
python -m build
python -m pip install dist/*.whl
```
Empty file removed docs/zh/custom_op_mapper.md
Empty file.

0 comments on commit 83ecf73

Please sign in to comment.