[QNN] Conv2D operator #3580

anijain2305 · 2019-07-18T20:54:57Z

Lowering of QNN Conv2D operation. We break the convolution into 4 terms as described in Option 1 here. Other relevant discussion is present at #2351

anijain2305 · 2019-07-18T20:56:12Z

cc @FrozenGene @tqchen @yzhliu

anijain2305 · 2019-08-02T18:55:18Z

src/relay/qnn/op/convolution.cc

+  }
+}
+
+/*


@FrozenGene @jackwish @u99127

While the Requantize and Legalize pass are going through final details, it will be useful to prefetch and look at QNN conv2d lowering. Please review and let me know your comments.

zhenhuaw-me

Thank you for ping. Suggesting to have if {NHWC} elif {NCHW} else {assert} when handling different memory layout. And, I am not sure dividing the compute into 4 terms is optimization-friendly, but that's out of this PR scope I think.

ps. Only looked into part of this PR, will recheck when the requantize op were merged. Feel free to ping please.

docs/langref/relay_op.rst

zhenhuaw-me · 2019-08-05T07:05:11Z

python/tvm/relay/qnn/op/qnn.py

+    input_scale of the input quantized tensors. The zero point of the output
+    quantized tensor is 0. By default, the dtype of output is int32. Please also
+    refer to Requantize operator to understand how to scale back the int32
+    ouptut to (u)int8.


zhenhuaw-me · 2019-08-05T07:07:20Z

python/tvm/relay/qnn/op/qnn.py

+           out_dtype="int32"):
+    r"""Quantized 2D convolution.
+
+    This operator convolves quantized weight with quantized data. The scale of


Can it be convolves quantized data with quantized weight? I know they are basically the same though...

Yup, that sounds better.

zhenhuaw-me · 2019-08-05T07:08:08Z

python/tvm/relay/qnn/op/qnn.py

+    r"""Quantized 2D convolution.
+
+    This operator convolves quantized weight with quantized data. The scale of
+    the output quantized tensor is the product of the weight_scale and


Do we need to align the term weight and kernel to be one of them rather than mixed?

Done to kernel

zhenhuaw-me · 2019-08-05T07:12:57Z

src/relay/op/tensor/reduce.cc

@@ -33,36 +34,6 @@

 namespace tvm {
 namespace relay {
-


If we are moving code like this to headers, would it be better to have a dedicated PR which involves no extra functionality?

src/relay/qnn/op/convolution.cc

zhenhuaw-me · 2019-08-05T07:17:43Z

src/relay/qnn/op/convolution.cc

+
+  const auto in_shape = get_shape(0);
+  int batch_size, in_channels;
+  // NCHW layout


Do we need to check it? Maybe simply if NCHW else if NHWC else assert?

Personally, I'd like to discuss the layout handling a bit (won't block the PR). To me, I always prefer code like if condition 1: path 1; elif condition 2: path 2: else (condition 3): path 3 or assert, even if the input guarantees that condition 3 won't happen. I think the path 1 inside a if section is easier to read, and the last assert is to handle unexpected code typo. Anyway, that is out the scope of this PR :).

zhenhuaw-me · 2019-08-05T07:18:02Z

src/relay/qnn/op/convolution.cc

+
+  const auto kernel_shape = get_shape(1);
+  int out_channels, kernel_h, kernel_w;
+  // OIHW layout


similar to input layout handling.

zhenhuaw-me · 2019-08-05T07:20:50Z

src/relay/qnn/op/convolution.cc

+    Array<IndexExpr> pad_w({param->padding[1], param->padding[1]});
+
+    Array<Array<IndexExpr>> pad_width;
+    pad_width = {pad_n, pad_c, pad_h, pad_w};


Layout check and handling?

anijain2305 · 2019-08-09T00:48:17Z

@u99127 @FrozenGene @jackwish @tqchen This is ready for review.

anijain2305 · 2019-08-09T00:51:49Z

@tqchen qnn.conv2d shares infer type functionality with nn.conv2d. Therefore, I have transferred that piece of code from cc to a header file and converted it to template function. It can now take Conv2DAttrs, QnnConv2dAttrs. Let me know if that looks ok.

anijain2305 · 2019-08-12T02:20:52Z

Pinging again in case this was missed @u99127 @FrozenGene @jackwish @tqchen

zhenhuaw-me

CI reports that there is merge conflicts, so request changes still...

It is very glad that we are reaching this, the conditional optimization is really beneficial. Thank you for the impressive work @anijain2305 !

src/relay/qnn/op/convolution.cc

zhenhuaw-me · 2019-09-02T07:34:11Z

src/relay/qnn/op/convolution.cc

+  }
+  auto reduced_t3 = Sum(Cast(weight, Int(32)), axes_t3, false, false);
+
+  // Find the newshape depenging on NCHW/NHWC layout.


is it depending?

zhenhuaw-me · 2019-09-02T07:46:51Z

src/relay/qnn/op/convolution.cc

+
+  const auto in_shape = get_shape(0);
+  int batch_size, in_channels;
+  // NCHW layout


Personally, I'd like to discuss the layout handling a bit (won't block the PR). To me, I always prefer code like if condition 1: path 1; elif condition 2: path 2: else (condition 3): path 3 or assert, even if the input guarantees that condition 3 won't happen. I think the path 1 inside a if section is easier to read, and the last assert is to handle unexpected code typo. Anyway, that is out the scope of this PR :).

anijain2305 · 2019-09-03T05:28:03Z

@jackwish Thanks for the good words :) I incorporated your comments. Can you please review again?

zhenhuaw-me

LGTM. Glad to participate in this, thank you for the great work!

src/relay/qnn/util.h

anijain2305 · 2019-09-03T16:11:28Z

@zhiics @vinx13 As jackwish has approved, can you please review?

zhiics

Overall LGTM, only left nit comment.

zhiics · 2019-09-03T21:05:50Z

src/relay/pass/pattern_util.h

@@ -415,6 +415,71 @@ static inline Expr Full(Expr fill_value,
  return CallNode::make(op, {fill_value}, Attrs(attrs), {});
 }

+static inline Expr Conv2D(Expr data, Expr weight, Array<IndexExpr> strides,


It looks this is the same as MakeConv2d, right?
If this is true, should we just keep one signature instead of having duplication. I am not strongly against this because it's obviously used by other cases as well.

Yeah, I kept it to follow other usecases. I guess this repetition might be because typically TVM does not want header and implementation linking problems. Will keep it Conv2D for now.

I don't have a strong feeling about this, actually I'm not sure why we prefer to copying this (and the others) here instead of adding declarations

I think linking should be fine. We can put TVM_DLL if needed. But anyway, we can keep it this way for now.

zhiics · 2019-09-03T21:06:46Z

src/relay/qnn/op/convolution.cc

+
+/*!
+ *  Copyright (c) 2019 by Contributors
+ * \file nn.cc


wrong file name.

zhiics · 2019-09-03T21:08:08Z

src/relay/qnn/op/convolution.cc

+ */
+WorkloadType GetWorkload(const Array<tvm::relay::Type>& arg_types, const QnnConv2DAttrs* param) {
+  // Get conv parameters.
+  auto get_shape = [&](const Type& type) {


we don't actually need to capture anything here, right?

zhiics · 2019-09-03T21:12:10Z

src/relay/qnn/op/convolution.cc

+  // Since, this is integer division (floor), we can first multiply the data by the pool_size and
+  // then perform avg_pool2d. Reversing this causes inaccuracy due to floor division.
+  auto scaled_hw_t2 = Multiply(casted_t2, MakeConstantScalar(Int(32), kernel_h * kernel_w));
+  Array<IndexExpr> padding;


Can we just use?

Array<IndexExpr> padding({0, 0})

zhiics · 2019-09-03T21:14:58Z

tests/python/relay/test_qnn_conv2d.py

+from tvm.relay.testing import create_workload
+from tvm.contrib import graph_runtime
+
+def run_infer_type(expr):


run_infer_type could be obtained from tvm.relay.testing as well.

vinx13 · 2019-09-04T00:29:50Z

src/relay/pass/pattern_util.h

@@ -415,6 +415,71 @@ static inline Expr Full(Expr fill_value,
  return CallNode::make(op, {fill_value}, Attrs(attrs), {});
 }

+static inline Expr Conv2D(Expr data, Expr weight, Array<IndexExpr> strides,


I don't have a strong feeling about this, actually I'm not sure why we prefer to copying this (and the others) here instead of adding declarations

tests/python/relay/test_qnn_conv2d.py

Rebasing. Empty commit. Clang-format styling.

zhiics

LGTM

Rebasing. Empty commit. Clang-format styling.

anijain2305 changed the title ~~[QNN] Convolution 2D Implementation.~~ [QNN] Conv2D operator Jul 18, 2019

anijain2305 mentioned this pull request Jul 18, 2019

[Relay] [Quantization] WIP - Protoyping the quantized convolution op #3367

Closed

anijain2305 force-pushed the qnn_conv2d branch from ed1195a to 6f83532 Compare July 18, 2019 21:10

anijain2305 changed the title ~~[QNN] Conv2D operator~~ [QNN] WIP - Conv2D operator Jul 18, 2019

anijain2305 force-pushed the qnn_conv2d branch 3 times, most recently from 4f6e6bf to 52a3c2b Compare July 23, 2019 23:29

anijain2305 force-pushed the qnn_conv2d branch from 52a3c2b to 63cf319 Compare July 31, 2019 22:54

anijain2305 commented Aug 2, 2019

View reviewed changes

anijain2305 changed the title ~~[QNN] WIP - Conv2D operator~~ [QNN] Conv2D operator Aug 2, 2019

anijain2305 force-pushed the qnn_conv2d branch 7 times, most recently from c35d314 to 020c123 Compare August 7, 2019 22:26

zhenhuaw-me suggested changes Aug 8, 2019

View reviewed changes

anijain2305 force-pushed the qnn_conv2d branch 6 times, most recently from bc5b780 to 699e34c Compare August 9, 2019 00:46

anijain2305 force-pushed the qnn_conv2d branch from 699e34c to 03cc5a1 Compare August 9, 2019 21:36

anijain2305 force-pushed the qnn_conv2d branch from 03cc5a1 to cadb1e5 Compare August 16, 2019 21:34

anijain2305 force-pushed the qnn_conv2d branch from cadb1e5 to 8baa53e Compare August 30, 2019 20:52

zhenhuaw-me suggested changes Sep 2, 2019

View reviewed changes

anijain2305 force-pushed the qnn_conv2d branch 2 times, most recently from f5cedbd to 434f40d Compare September 3, 2019 05:25

zhenhuaw-me approved these changes Sep 3, 2019

View reviewed changes

src/relay/qnn/util.h Show resolved Hide resolved

anijain2305 force-pushed the qnn_conv2d branch from 434f40d to 7773c32 Compare September 3, 2019 16:10

zhiics reviewed Sep 3, 2019

View reviewed changes

anijain2305 force-pushed the qnn_conv2d branch from 7773c32 to f2e7ec4 Compare September 4, 2019 00:20

vinx13 approved these changes Sep 4, 2019

View reviewed changes

[QNN] Convolution 2D Implementation.

d3a54c2

Rebasing. Empty commit. Clang-format styling.

anijain2305 force-pushed the qnn_conv2d branch from f2e7ec4 to d3a54c2 Compare September 4, 2019 01:36

zhiics approved these changes Sep 4, 2019

View reviewed changes

zhiics merged commit 0d4870c into apache:master Sep 4, 2019

zhiics added the status: accepted label Sep 4, 2019

This was referenced Sep 5, 2019

Do type checking for the input and kernel in the qnn conv2d #3904

Merged

Qnn fully connected #3910

Merged

MarisaKirisame pushed a commit to MarisaKirisame/tvm that referenced this pull request Sep 7, 2019

[QNN] Convolution 2D Implementation. (apache#3580)

9464962

Rebasing. Empty commit. Clang-format styling.

wweic pushed a commit to wweic/tvm that referenced this pull request Sep 16, 2019

[QNN] Convolution 2D Implementation. (apache#3580)

c3e5bef

Rebasing. Empty commit. Clang-format styling.

wweic pushed a commit to wweic/tvm that referenced this pull request Sep 16, 2019

[QNN] Convolution 2D Implementation. (apache#3580)

687f8ab

Rebasing. Empty commit. Clang-format styling.

wweic pushed a commit to neo-ai/tvm that referenced this pull request Sep 16, 2019

[QNN] Convolution 2D Implementation. (apache#3580)

9cf6590

Rebasing. Empty commit. Clang-format styling.

tqchen mentioned this pull request Nov 8, 2019

[RELEASE][DRAFT] TVM v0.6 Release candidate #4259

Closed

+                }
+              }
+              /*

[QNN] Conv2D operator #3580

[QNN] Conv2D operator #3580

Conversation

anijain2305 commented Jul 18, 2019 • edited Loading

anijain2305 commented Jul 18, 2019 • edited Loading

Choose a reason for hiding this comment

zhenhuaw-me left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anijain2305 commented Aug 9, 2019

anijain2305 commented Aug 9, 2019

anijain2305 commented Aug 12, 2019

zhenhuaw-me left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anijain2305 commented Sep 3, 2019

zhenhuaw-me left a comment

Choose a reason for hiding this comment

anijain2305 commented Sep 3, 2019

zhiics left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhiics left a comment

Choose a reason for hiding this comment

anijain2305 commented Jul 18, 2019 •

edited

Loading

anijain2305 commented Jul 18, 2019 •

edited

Loading