[5/6] Arm(R) Ethos(TM)-U NPU codegen integration #8849

manupak · 2021-08-25T17:49:19Z

This commit integrates the codegen for Arm® Ethos™-U.

Adding Conv2D tests and a mobilenet_v1 conv2d offload test.

This PR is blocked on merging #8795 , #8806 and #8811. Since this is built on top of all those PR branches, this contains the accumulation of contents of the all the PRs.

Co-authored-by: Grant Watson [email protected]
Co-authored-by: Leandro Nunes [email protected]
Co-authored-by: Christopher Sidebottom [email protected]
Co-authored-by: Matthew Barrett [email protected]

manupak · 2021-09-13T18:02:38Z

@Mousius @grant-arm , #8274 fails test_networks.py . I cant immediately figure out why. See if you guys see something.

cc: @mbaret

manupak · 2021-09-13T18:24:18Z

From the looks of it,

TVMBAW calls are not generated any more for static allocates for kDLCPU. I wonder why the AoT tests did not break.

grant-arm · 2021-09-13T20:30:47Z

src/relay/backend/contrib/ethosu/source_module.cc

+   */
+  void SaveToFile(const std::string& file_name, const std::string& format) final {
+    std::string fmt = GetFileFormat(file_name, format);
+    LOG(INFO) << "format=" << fmt << ";;\n";


This looks like some debug logging that was left in by mistake.
It should probably be removed.

jroesch · 2021-09-15T18:00:27Z

src/tir/transforms/lower_tvm_builtin.cc

@@ -113,16 +113,6 @@ class BuiltinLower : public StmtExprMutator {
    op = stmt.as<AllocateNode>();
    // Get constant allocation bound.
    int64_t nbytes = GetVectorBytes(op->dtype);
-    if (device_type_.defined()) {


cc @areusch @mbrookhart this is the code we discussed yesterday, currently CI passes with this code? that seems to contradict the conversations we had, but also I think these should be stack allocations for performance on cloud cpus.

Yes, if we want the allocates to be placed on stack in CPU PrimFuncs, maybe we should make them have storage_scope = 'local' and we should generate TVMBAWs for 'global' allocates -- that could be made to work for both cases.

Filed #9022.
Just checking your use case: You want to see the allocas so you can later rewrite them, right? It's a bit confusing because I thought if the final c code generator sees these Allocates it rewrites them to use globals which I assumed was what you wanted.

grant-arm · 2021-09-16T15:35:09Z

src/relay/backend/contrib/ethosu/source_module.cc

+
+    ss << "#include <stdio.h>\n";
+    ss << "#include <stdlib.h>\n";
+    ss << "#include <dlpack/dlpack.h>\n";


dlpack.h is no longer required.

This commit integrates the codegen for Arm® Ethos™-U. * Adding Conv2D tests and a mobilenet_v1 conv2d offload test. Co-authored-by: Grant Watson <[email protected]> Co-authored-by: Leandro Nunes <[email protected]> Co-authored-by: Christopher Sidebottom <[email protected]> Co-authored-by: Matthew Barret <[email protected]>

ekalda · 2021-09-28T15:32:02Z

tests/python/contrib/test_ethosu/infra.py

+            module=module,
+            inputs=inputs,
+            outputs=outputs,
+            output_tolerance=10,


I think the default value of output_tolerance should be 0 and build_source should have an optional parameter to set it to something else, if necessary

mbs-octoml

LGTM modulo some nits and comment requests.

I didn't dig into the tests much, esp the reference_system stuff, will trust your local reviewers.

mbs-octoml · 2021-09-28T19:19:58Z

src/relay/backend/contrib/ethosu/source_module.cc

+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <dmlc/filesystem.h>


nit: file level preamble comment

mbs-octoml · 2021-09-28T19:20:41Z

src/relay/backend/contrib/ethosu/source_module.cc

+  /*!
+   * \brief The ethos runtime module.
+   *
+   * \param cmms A array of external symbol 1, serialized command stream 1


nit: comment seems garbled, might as well explain them all here or on the fields.

mbs-octoml · 2021-09-28T19:24:06Z

src/relay/backend/contrib/ethosu/source_module.cc

+namespace tvm {
+namespace runtime {
+
+class EthosUModuleNode : public ModuleNode {


Overview comment, especially since this is doing the final host codegen.

mbs-octoml · 2021-09-28T19:29:18Z

src/relay/backend/contrib/ethosu/source_module.cc

+
+ private:
+  String c_source;
+  Array<String> func_names_;


Worth changing to a single main_func_name_ since a deep assumption throughout?

mbs-octoml · 2021-09-28T19:34:41Z

python/tvm/relay/backend/contrib/ethosu/codegen.py

+
+
+@tvm._ffi.register_func("relay.ext.ethosu.constant_updater")
+def constant_updater(expr, symbol):  # pylint: disable=unused-argument


nit: move below main entry point and comment it is called back from utils.h UpdateConstants after lowering.

mbs-octoml · 2021-09-28T19:35:48Z

src/relay/backend/contrib/ethosu/source_module.cc

+}
+
+TVM_REGISTER_GLOBAL("runtime.module.ethosu.create").set_body([](TVMArgs args, TVMRetValue* rv) {
+  *rv = EthosUModuleNode::Create(args[0], args[1], args[2], args[3], args[4], args[5]);


nit: consider TypedPackedFunc

jroesch · 2021-09-28T23:43:54Z

python/tvm/relay/backend/contrib/ethosu/codegen.py

+
+
+@tvm._ffi.register_func("relay.ext.ethosu")
+def ethosu_compiler(ref):


Update argument name/improve doc string here?

jroesch

Minor nits, otherwise mostly Ethos-U specific and looks good to me

jroesch · 2021-09-28T23:45:56Z

tests/python/contrib/test_ethosu/__init__.py

+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Test infrastructure for Arm(R) Ethos(TM)-U NPU related tests"""


Nit: grammar

manupak · 2021-09-29T08:36:38Z

Thanks all!
I'll address the minors in a follow up (as updated here : #8482).

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

* main: Fix flaky NMS test by making sure scores are unique (apache#9140) [Relay] Merge analysis/context_analysis.cc and transforms/device_annotation.cc (apache#9038) [LLVM] Make changes needed for opaque pointers (apache#9138) Arm(R) Ethos(TM)-U NPU codegen integration (apache#8849) [CI] Split Integration tests out of first phase of pipeline (apache#9128) [Meta Schedule][M3b] Runner (apache#9111) Fix Google Mock differences between Ubuntu 18.04 and 16.04 (apache#9141) [TIR] add loop partition hint pragma (apache#9121) fix things (apache#9146) [Meta Schedule][M3a] SearchStrategy (apache#9132) [Frontend][PyTorch] support for quantized conv_transpose2d op (apache#9133) [UnitTest] Parametrized test_conv2d_int8_intrinsics (apache#9143) [OpenCL] Remove redundant visit statement in CodeGen. (apache#9144) [BYOC] support arbitrary input dims for add/mul/relu of dnnl c_src codegen (apache#9127) [Relay][ConvertLayout] Support for qnn.conv2d_transpose (apache#9139) add nn.global_avgpool to fq2i (apache#9137) [UnitTests] Enable minimum testing on Vulkan target in CI (apache#9093) [Torch] Support returning quantized weights and bias for BYOC use cases (apache#9135) [Relay] Prepare for new plan_devices.cc (part II) (apache#9130) [microTVM][Zephyr] Add MIMXRT1050 board support (apache#9068)

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

* main: (80 commits) Introduce centralised name transformation functions (apache#9088) [OpenCL] Add vectorization to cuda conv2d_nhwc schedule (apache#8636) [6/6] Arm(R) Ethos(TM)-U NPU codegen integration with `tvmc` (apache#8854) [microTVM] Add wrapper for creating project using a MLF (apache#9090) Fix typo (apache#9156) [Hotfix][Testing] Wait for RPCServer to be established (apache#9150) Update find cublas so it search default path if needed. (apache#9149) [TIR][LowerMatchBuffer] Fix lowering strides when source region has higher dimension than the buffer (apache#9145) Fix flaky NMS test by making sure scores are unique (apache#9140) [Relay] Merge analysis/context_analysis.cc and transforms/device_annotation.cc (apache#9038) [LLVM] Make changes needed for opaque pointers (apache#9138) Arm(R) Ethos(TM)-U NPU codegen integration (apache#8849) [CI] Split Integration tests out of first phase of pipeline (apache#9128) [Meta Schedule][M3b] Runner (apache#9111) Fix Google Mock differences between Ubuntu 18.04 and 16.04 (apache#9141) [TIR] add loop partition hint pragma (apache#9121) fix things (apache#9146) [Meta Schedule][M3a] SearchStrategy (apache#9132) [Frontend][PyTorch] support for quantized conv_transpose2d op (apache#9133) [UnitTest] Parametrized test_conv2d_int8_intrinsics (apache#9143) ...

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

This is a follow up commit to address the comments of #8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

This commit integrates the codegen for Arm® Ethos™-U. * Adding Conv2D tests and a mobilenet_v1 conv2d offload test. Co-authored-by: Grant Watson <[email protected]> Co-authored-by: Leandro Nunes <[email protected]> Co-authored-by: Christopher Sidebottom <[email protected]> Co-authored-by: Matthew Barret <[email protected]> Co-authored-by: Grant Watson <[email protected]> Co-authored-by: Leandro Nunes <[email protected]> Co-authored-by: Christopher Sidebottom <[email protected]> Co-authored-by: Matthew Barret <[email protected]>

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

This commit integrates the codegen for Arm® Ethos™-U. * Adding Conv2D tests and a mobilenet_v1 conv2d offload test. Co-authored-by: Grant Watson <[email protected]> Co-authored-by: Leandro Nunes <[email protected]> Co-authored-by: Christopher Sidebottom <[email protected]> Co-authored-by: Matthew Barret <[email protected]> Co-authored-by: Grant Watson <[email protected]> Co-authored-by: Leandro Nunes <[email protected]> Co-authored-by: Christopher Sidebottom <[email protected]> Co-authored-by: Matthew Barret <[email protected]>

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

manupak requested review from anijain2305, areusch, comaniac, jroesch, junrushao, jwfromm, MarisaKirisame, mbaret, mbrookhart, merrymercy, slyubomirsky, tqchen, trevor-m, vinx13, wweic, yzhliu, zhiics, ZihengJiang and a team as code owners August 25, 2021 17:49

manupak marked this pull request as draft August 25, 2021 17:49

manupak mentioned this pull request Aug 25, 2021

[RFC][Tracking Issue] Arm® Ethos™-U Integration #8482

Closed

8 tasks

mbs-octoml mentioned this pull request Aug 25, 2021

Implementation of relay_to_tir target hook #8423

Merged

leandron mentioned this pull request Aug 26, 2021

[6/6] Arm(R) Ethos(TM)-U NPU codegen integration with tvmc #8854

Merged

manupak force-pushed the ethosu_upstreaming_p5 branch from 1e08789 to d6d8bec Compare August 26, 2021 11:24

Mousius force-pushed the ethosu_upstreaming_p5 branch 3 times, most recently from 7522dda to cd1a9c8 Compare August 26, 2021 18:30

grant-arm mentioned this pull request Sep 3, 2021

Arm(R) Cortex(R)-M55 CPU and Arm(R) Ethos(TM)-U55 NPU Demo App #8922

Merged

grant-arm reviewed Sep 13, 2021

View reviewed changes

manupak force-pushed the ethosu_upstreaming_p5 branch from cd1a9c8 to 0523597 Compare September 14, 2021 15:23

jroesch reviewed Sep 15, 2021

View reviewed changes

grant-arm reviewed Sep 16, 2021

View reviewed changes

manupak force-pushed the ethosu_upstreaming_p5 branch from 7824786 to 962a7ff Compare September 25, 2021 08:05

manupak marked this pull request as ready for review September 27, 2021 19:38

manupak force-pushed the ethosu_upstreaming_p5 branch from 962a7ff to 4f67889 Compare September 27, 2021 19:48

ekalda reviewed Sep 28, 2021

View reviewed changes

mbs-octoml approved these changes Sep 28, 2021

View reviewed changes

jroesch reviewed Sep 28, 2021

View reviewed changes

jroesch approved these changes Sep 28, 2021

View reviewed changes

manupak added a commit to manupak/incubator-tvm that referenced this pull request Sep 29, 2021

Cleaning up Arm(R) Ethos(TM)-U codegen

3a3fe0a

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

manupak mentioned this pull request Sep 29, 2021

A follow up PR for 5/6 of Arm(R) Ethos(TM)-U NPU codegen #9147

Merged

jroesch merged commit 86ce111 into apache:main Sep 29, 2021

Mousius deleted the ethosu_upstreaming_p5 branch September 29, 2021 16:54

manupak added a commit to manupak/incubator-tvm that referenced this pull request Sep 30, 2021

Cleaning up Arm(R) Ethos(TM)-U codegen

6b96adf

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

manupak added a commit to manupak/incubator-tvm that referenced this pull request Sep 30, 2021

Cleaning up Arm(R) Ethos(TM)-U codegen

98ec6f1

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

manupak added a commit to manupak/incubator-tvm that referenced this pull request Sep 30, 2021

Cleaning up Arm(R) Ethos(TM)-U codegen

52e570d

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

manupak added a commit to manupak/incubator-tvm that referenced this pull request Oct 1, 2021

Cleaning up Arm(R) Ethos(TM)-U codegen

b297d67

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

jroesch pushed a commit that referenced this pull request Oct 1, 2021

Cleaning up Arm(R) Ethos(TM)-U codegen (#9147)

4f6b2a1

This is a follow up commit to address the comments of #8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

ylc pushed a commit to ylc/tvm that referenced this pull request Jan 7, 2022

Cleaning up Arm(R) Ethos(TM)-U codegen (apache#9147)

81fa719

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

ylc pushed a commit to ylc/tvm that referenced this pull request Jan 13, 2022

Cleaning up Arm(R) Ethos(TM)-U codegen (apache#9147)

25c1501

This is a follow up commit to address the comments of apache#8849 Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[5/6] Arm(R) Ethos(TM)-U NPU codegen integration #8849

[5/6] Arm(R) Ethos(TM)-U NPU codegen integration #8849

manupak commented Aug 25, 2021 •

edited

Loading

manupak commented Sep 13, 2021 •

edited

Loading

manupak commented Sep 13, 2021 •

edited

Loading

grant-arm Sep 13, 2021

jroesch Sep 15, 2021

manupak Sep 15, 2021 •

edited

Loading

mbs-octoml Sep 15, 2021 •

edited

Loading

grant-arm Sep 16, 2021

ekalda Sep 28, 2021

mbs-octoml left a comment

mbs-octoml Sep 28, 2021

mbs-octoml Sep 28, 2021

mbs-octoml Sep 28, 2021

mbs-octoml Sep 28, 2021

mbs-octoml Sep 28, 2021

mbs-octoml Sep 28, 2021

jroesch Sep 28, 2021

jroesch left a comment

jroesch Sep 28, 2021

manupak commented Sep 29, 2021 •

edited

Loading



		@tvm._ffi.register_func("relay.ext.ethosu.constant_updater")
		def constant_updater(expr, symbol): # pylint: disable=unused-argument



		@tvm._ffi.register_func("relay.ext.ethosu")
		def ethosu_compiler(ref):

[5/6] Arm(R) Ethos(TM)-U NPU codegen integration #8849

[5/6] Arm(R) Ethos(TM)-U NPU codegen integration #8849

Conversation

manupak commented Aug 25, 2021 • edited Loading

manupak commented Sep 13, 2021 • edited Loading

manupak commented Sep 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manupak Sep 15, 2021 • edited Loading

Choose a reason for hiding this comment

mbs-octoml Sep 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbs-octoml left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jroesch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manupak commented Sep 29, 2021 • edited Loading

manupak commented Aug 25, 2021 •

edited

Loading

manupak commented Sep 13, 2021 •

edited

Loading

manupak commented Sep 13, 2021 •

edited

Loading

manupak Sep 15, 2021 •

edited

Loading

mbs-octoml Sep 15, 2021 •

edited

Loading

manupak commented Sep 29, 2021 •

edited

Loading