Asp movement #36525

mingxu1067 · 2021-10-19T02:50:25Z

PR types

Others

PR changes

APIs

Describe

Added Automatic SParsity (ASP)'s alias into paddel.static.

Since the folder organization changes in v2.2, the paddle.fluid.xxx is not the official way to use modules. We move ASP module to paddle.static to make its usage be consistent with official ways, also modified code examples and related unittests.

Example:

import paddle
import paddle.fluid as fluid
import paddle.fluid.core as core

# Previous Usage
# from paddle.fluid.contrib import sparsity
# Current Usage
from paddle.static import sparsity

# Others are the same as previous.
paddle.enable_static()

main_program = paddle.static.Program()
startup_program = paddle.static.Program()

with paddle.static.program_guard(main_program, startup_program):
	input_data = paddle.static.data(name='data', shape=[None, 128])
	label = paddle.static.data(name='label', shape=[None, 10])
	hidden = paddle.static.nn.fc(x=input_data, num_flatten_dims=-1, size=32, activation=None, name="need_sparse_fc")
	hidden = paddle.static.nn.fc(x=hidden, num_flatten_dims=-1, size=32, activation=None, name="need_dense_fc")
	prob = paddle.static.nn.fc(x=hidden, num_flatten_dims=-1, size=10, activation=None)
	loss = paddle.mean(paddle.nn.functional.square_error_cost(prob, label))

	# Setup exluded layers out from ASP workflow.
	# Please note, excluded_layers must be set before calling `optimizer.minimize()`.
	sparsity.set_excluded_layers(main_program, ["need_dense_fc"])

	optimizer = paddle.optimizer.SGD(learning_rate=0.1)
	optimizer = paddle.static.amp.decorator.decorate(optimizer )
	# Calling sparsity.decorate() to wrap minimize() in optimizer, which 
	# will insert necessary masking operations for ASP workflow.
	optimizer = sparsity.decorate(optimizer)
	optimizer.minimize(loss, startup_program)

device = paddle.device.get_device()
place  = paddle.set_device(device)

exe = paddle.static.Executor(place)
exe.run(startup_program)

# Must call `exe.run(startup_program)` first before calling `sparsity.prune_model`
sparsity.prune_model(place, main_program, func_name=sparsity.MaskAlgo.MASK_2D_BEST)

# Do training loop

…modified examples in doc.

… asp_movement

Xreki · 2021-10-26T02:23:54Z

python/paddle/fluid/contrib/sparsity/asp.py

@@ -66,7 +66,7 @@ def decorate(optimizer):

            import paddle
            import paddle.fluid as fluid
-            from paddle.fluid.contrib import sparsity


需要对用户暴露的接口的示例代码中，不要再import fluid，也不要用任何fluid的API。

好的已經全面修正

Xreki · 2021-10-26T02:56:44Z

python/paddle/fluid/contrib/sparsity/asp.py

@@ -158,6 +154,8 @@ def prune_model(place,
                optimizer = sparsity.decorate(optimizer)
                optimizer.minimize(loss, startup_program)

+            place = paddle.CPUPlace()


为什么要设置CPUPlace？

修改成使用paddle.get_device來判斷Place的裝置

Xreki · 2021-10-26T02:58:48Z

python/paddle/static/sparsity/__init__.py

+from ...fluid.contrib.sparsity import create_mask  #noqa: F401
+from ...fluid.contrib.sparsity import check_sparsity  #noqa: F401
+from ...fluid.contrib.sparsity import MaskAlgo  #noqa: F401
+from ...fluid.contrib.sparsity import CheckMethod  #noqa: F401


再确认一下，以上API都需要向用户暴露吗？

一定需要暴露給用戶的API僅為MaskAlgo，目的是要讓用戶能夠決定Pruning的方式 (1D, 2D_greedy, 2D_best)等。
其餘依照建議移除

XiaoguangHu01 · 2021-10-28T12:31:36Z

python/paddle/static/sparsity/__init__.py

+from ...fluid.contrib.sparsity import decorate  #noqa: F401
+from ...fluid.contrib.sparsity import prune_model  #noqa: F401
+from ...fluid.contrib.sparsity import set_excluded_layers  #noqa: F401
+from ...fluid.contrib.sparsity import reset_excluded_layers  #noqa: F401


将需要对外公开的API 放到__all__列表，paddle会根据__all__列表区别公开API和内部使用API，
公开API需要提供使用文档和示例

新增__all__設定
相關文件與範例撰寫於
https://github.com/mingxu1067/Paddle/blob/asp_movement/python/paddle/fluid/contrib/sparsity/asp.py
https://github.com/mingxu1067/Paddle/blob/asp_movement/python/paddle/fluid/contrib/sparsity/utils.py

XiaoguangHu01 · 2021-10-28T12:37:48Z

python/paddle/static/sparsity/__init__.py

+# limitations under the License.
+
+from ...fluid.contrib.sparsity import calculate_density  #noqa: F401
+from ...fluid.contrib.sparsity import MaskAlgo  #noqa: F401


MaskAlgo这个类型可以作为内部使用的数据类型使用，不建议对外公开。
比如这种写法暴露了较多内部实现的细节，书写和阅读都不是很方便
prune_model(... func_name=sparsity.MaskAlgo.MASK_1D)
建议改成这种形式更简洁，减少对外公开的数据类型，采用小写方便输入
prune_model(... mask_algo='mask_1d')

採用建議使用小寫string作為輸入選項，並隱藏 MaskAlgo

XiaoguangHu01 · 2021-10-28T12:43:46Z

python/paddle/fluid/contrib/sparsity/asp.py

@@ -114,7 +112,7 @@ def prune_model(place,
    inference only. To obtain OptimizerWithSparsityGuarantee, please see `sparsity.decoreate()`.

    Args:
-        place (fluid.CPUPlace()|fluid.CUDAPlace(N)): Device place for pruned parameter and mask Variables, and N means the GPU's id. It should be the same as created instance of Executor.
+        place (paddle.CPUPlace()|paddle.CUDAPlace(N)): Device place for pruned parameter and mask Variables, and N means the GPU's id. It should be the same as created instance of Executor.


确认下，place这个参数是否是必须的？
是否可以用这两组参数来修改全局的place信息？
paddle.device.get_device()
paddle.device.set_device()

可以的，依照建議移除place argument 採用paddle.device.get_device(), paddle.device.set_device()來自動判斷

1. Do not expose MaskAlgo, instead using string as data type of func_name. 2. Removed 'place' arg in prune_model() 3. Added __all__ in paddle.static.sparsity

Xreki

LGTM

XiaoguangHu01 · 2021-10-29T03:21:55Z

python/paddle/fluid/contrib/sparsity/asp.py

                n=2,
                m=4,
-                func_name=sparsity.MaskAlgo.MASK_1D,
+                func_name='mask_1d',


这里api的参数名称func_name是否指mask algorithm？建议再改一下，更清晰一些
func_name -> mask_algo

依照建議修改完成。

XiaoguangHu01 · 2021-10-29T03:22:47Z

python/paddle/fluid/contrib/sparsity/asp.py

        main_program (Program, optional): Program with model definition and its parameters. Default is `paddle.static.default_main_program()
        n (int): n of `n:m` sparse pattern.
        m (int): m of `n:m` sparse pattern.
-        func_name (MaskAlgo, optional): The function name to generate spase mask. Default is `MaskAlgo.MASK_1D`. All options please refer to `MaskAlgo`.
+        func_name (string, optional): The function name to generate spase mask. Default is `mask_1d`.


XiaoguangHu01

LGTM

)

Cherry-pick #36525

)

mingxu1067 added 7 commits October 15, 2021 17:09

Added an alias of ASP into paddle.static for PaddlePaddle.static and …

da1ed48

…modified examples in doc.

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

f6dac96

… asp_movement

Added Nvidia Copyright

806d642

Added paddle.static.sparsity to setup.py.in

0e071cb

Code format fix

22336af

Fixed code example errors.

8dca054

Fixed code example errors in ASP.

5ee159f

Xreki reviewed Oct 26, 2021

View reviewed changes

mingxu1067 added 7 commits October 26, 2021 06:13

Replaced fluid.* to paddle.static in code examples

550e926

Removed some APIs alias, which users donot need to know.

b24c964

Fix example errors

cd95bd5

Modified ASP unittests for new APIs alias

66081ce

Fixed errors in sparsity example code.

175eba7

Fixed errors in sparsity example code.

1d3fc96

Fixed errors in test_fleet_with_asp_*

34c56f4

XiaoguangHu01 reviewed Oct 28, 2021

View reviewed changes

mingxu1067 added 2 commits October 28, 2021 15:29

Do not exposep MaskAlgo and some changes.

6ae246c

1. Do not expose MaskAlgo, instead using string as data type of func_name. 2. Removed 'place' arg in prune_model() 3. Added __all__ in paddle.static.sparsity

Fixed code example errors.

c39c5dc

Xreki previously approved these changes Oct 29, 2021

View reviewed changes

XiaoguangHu01 reviewed Oct 29, 2021

View reviewed changes

Rename argument func_name to mask_algo in ASP's pruning

225ce60

mingxu1067 dismissed Xreki’s stale review via 225ce60 October 29, 2021 06:27

XiaoguangHu01 approved these changes Oct 29, 2021

View reviewed changes

Xreki merged commit 113816d into PaddlePaddle:develop Oct 29, 2021

Xreki pushed a commit to Xreki/Paddle that referenced this pull request Oct 29, 2021

Move the ASP training API to paddle.static.sparsity. (PaddlePaddle#36525

552a431

)

Xreki mentioned this pull request Oct 29, 2021

[Cherry-pick 2.2] Move the ASP training API to paddle.static.sparsity. (#36525) #36860

Merged

lanxianghit pushed a commit that referenced this pull request Oct 29, 2021

Move the ASP training API to paddle.static.sparsity. (#36525) (#36860)

09bc9c0

Cherry-pick #36525

ghost pushed a commit to piotrekobi/Paddle that referenced this pull request Nov 3, 2021

Move the ASP training API to paddle.static.sparsity. (PaddlePaddle#36525

8f1938d

)

mingxu1067 deleted the asp_movement branch December 17, 2021 09:31

mingxu1067 added the NVIDIA label Jan 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Asp movement #36525

Asp movement #36525

mingxu1067 commented Oct 19, 2021 •

edited

Loading

Xreki Oct 26, 2021

mingxu1067 Oct 26, 2021

Xreki Oct 26, 2021

mingxu1067 Oct 26, 2021

Xreki Oct 26, 2021

mingxu1067 Oct 26, 2021

XiaoguangHu01 Oct 28, 2021

mingxu1067 Oct 28, 2021

XiaoguangHu01 Oct 28, 2021

mingxu1067 Oct 28, 2021

XiaoguangHu01 Oct 28, 2021

mingxu1067 Oct 28, 2021

Xreki left a comment

XiaoguangHu01 Oct 29, 2021

mingxu1067 Oct 29, 2021

XiaoguangHu01 Oct 29, 2021

XiaoguangHu01 left a comment

Asp movement #36525

Asp movement #36525

Conversation

mingxu1067 commented Oct 19, 2021 • edited Loading

PR types

PR changes

Describe

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xreki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

mingxu1067 commented Oct 19, 2021 •

edited

Loading