Fix TensorRT integration and add TRT tests to CI #41

trevor-m · 2019-08-28T22:23:29Z

There was two issues preventing the TensorRT integration from working. These regressions seem to have been caused by merges from upstream (this commit) which accidentally overwrote the original TRT integration code.

nnvm/python/nnvm/compiler/build_module.py: The target param was no longer being passed to optimize. The conditions in optimize would therefore always prevent TRT subgraphs from being partitioned. I have removed these extra conditions so they are not needed anyway.
nnvm/src/compiler/graph_compile.cc: When the entire graph is converted to one TRT subgraph, there is nothing left for nnvm to compile into a module. In that situation, nnvm.compiler.build_target would hit an assertion and cause a crash. So instead we create an empty module.

To prevent this from occurring again in the future, I've added the existing TRT tests to the CI. I had to create a new docker image with TRT in order to do this, currently it is located here and the dockerfile is at docker/Dockerfile.ci_gpu_trt.

Other changes:

Added a check in tests to make sure TRT subgraphs are created.
Improved TensorRT.cmake to search for system install in additional to manually specified path.

trevor-m · 2019-08-29T21:19:59Z

@reminisce Could you please review? Thanks!

reminisce · 2019-08-29T21:59:28Z

nnvm/python/nnvm/compiler/build_module.py

@@ -211,11 +211,7 @@ def optimize(graph, shape, dtype="float32", layout=None, target=None):
        graph = graph.apply(["CorrectLayout"])

    if cfg.ext_accel and cfg.ext_accel == 'tensorrt':
-        if isinstance(target, dict):


I think you still need to keep this logic to avoid partitioning graphs in PrecomputePrune pass where target is llvm.

Thanks for reviewing! Could you explain this more?

The ext_accel would still have to be set to tensorrt for partitioning to happen, so it doesn't seem like a problem to me. If you set ext_accel=tensorrt, you always want partitioning to happen right?

PrecomputePrune doesn't seem to call the optimize function, how is it related here?

Why is the target required to be cuda for tensorrt?

Thanks!

trevor-m · 2019-10-15T17:20:16Z

I removed some of the changes since NNVM-TRT is considered legacy now.

yongwww · 2019-10-15T17:59:59Z

@reminisce @zhiics @wweic

zhiics · 2019-10-15T18:06:24Z

Looks good to me. Please fix the ci.

trevor-m · 2019-10-15T18:10:25Z

CI error looks like some unrelated problem with Jenkins @hcho3

+ docker/bash.sh tvmai/ci-lint:v0.51 ./tests/scripts/task_lint.sh

WORKSPACE: /home/ubuntu/workspace/tvm/sanity

DOCKER CONTAINER NAME: tvmai/ci-lint:v0.51


Running './tests/scripts/task_lint.sh' inside tvmai/ci-lint:v0.51...

docker

Unable to find image 'tvmai/ci-lint:v0.51' locally

docker: Error response from daemon: received unexpected HTTP status: 502 Bad Gateway.

See 'docker run --help'.

script returned exit code 125

yongwww

LGTM

Remove unnecessary target param and conditions from BuildConfig.optimize(). Since target wasn't getting passed to optimize() from build(), TRT subgraph partioning was skipped due to unnecessary conditions. Create empty module when there are no funcs to compile to avoid crash Update TensorRT tests and add to CI Fix bug in TRT cmake Remove changes to CI

trevor-m · 2019-12-17T19:01:15Z

@ziyu-guo CI is passing now.

…generating (apache#5962) * Code migration Start (neo-ai#1) * Init commit: Code migration Start * Add loop_state.cc/h * Add ComputeDAG basic test * Split transform_step out & Update more UTs (neo-ai#3) * Split transform_step out * Update GetProducers & GetConsumers * Update UTs * Add UT for CacheReadWrite & Some bug fix * Add search_task, measure and serialization (neo-ai#4) * Add FollowSplit & FollowFusedSplit tests * Update dag.InferBound & its UT * Add search_task, measure and serialization * Update Serialization UT * Add MetaTileRewritePolicy (neo-ai#5) * Add feature * Add cost_model, meta_tile_rewrite_policy * Add MetaTileRewritePolicy basic UT * Basic Python API for State (neo-ai#6) * Add Basic Python API for State * Add UTs for State * Add Python API: Measure & Task (neo-ai#7) * Update the return value of state operation * Add task * Copy measure.py & utils.py * Fix LocalBuilder * Fix LocalRunner * Add ansor.auto_schedule() API; First AutoSchedule working version(neo-ai#8) * Add basic Python support for ansor.auto_schedule * Update AutoSchedule API * Bug fix for get the attach point of a fused iter * Update UT after infer bug fix * Bug fix & Add python serialization API (neo-ai#10) * Delete C++ UT hack since Python is ready * Add ndarray.non_empty * Update Serialization python API * Improve code style, python wrapper and test cases (neo-ai#11) * Update c++ code style and unit test * Update python State wrapper and test cases * fix unit tests * Add RPCRunner & OpenCL/CUDA test (neo-ai#12) * Add RPCRunner & OpenCL search test * Add CUDA search test * Add RPCRunner test * rebase to upstream/master * Add Ansor basic tutorial (neo-ai#13) * Add basic tutorial * migrate feature extraction (neo-ai#14) * Add XGBModel & RPCRunnerWarpper (neo-ai#15) * Add XGBModel & RPCRunnerWarpper * Revert "Add Parallel Granularity Mutation" * Migrate workload_registry.py (neo-ai#16) * add workload registry * update * update * add task scheduler (neo-ai#17) * Add conv2d cuda tutorial with workload registry (neo-ai#18) * add tune_test.py (the old tune_wkl.py) (neo-ai#19) * add tune_test.py (the old tune_wkl.py) * update * fix measure * fix for gpu * Code refine for tune_test.py & Add a pre load callback (neo-ai#20) * Bug fix for tutorials * Add PreLoadMeasuredStates * Add search_callback support for task tuner * Code refine for tune_test.py * Update * Update * Update * Update * Bug fix * Add python custom sketch rule (neo-ai#21) * Add custom sketch rule * Bug fix * Ansor Relay Integration (without layout rewrite) (neo-ai#22) * relay integration * Add tune_op_subgraph.py & Some code clean for tune_network.py (neo-ai#23) * Add single op tune scripts * Add tune subgraph support * Merge all op & all subgraph to one file * Rename file * add explicit_unroll_max_extent (neo-ai#25) * Add Index simplification & API update (neo-ai#26) * Add vectorized cooperative_fetching test * Update math simplify for vectorized CF * File rename * Update tune_network * API update * Update PreLoadMeasuredStates & Some bug fix (neo-ai#27) * Add a threading wrapper to fix the test bug * Set default TVM_USE_AUTO_SCHEDULER to false * Update PreLoadMeasuredStates callback * Add tensorize step for loop_state (neo-ai#31) * Add tensorize step * State python api update (neo-ai#33) * Start to update api * Add compute_dag to state * API update * kernel layout rewrite (neo-ai#28) * kernel layout rewrite * remove some hacks * add defuse_ops pass and move kernel_layout_rewrite pass after fuse_ops pass * set TVM_RELAY_DISABLE_BUILD_CACHE for task extraction and prepare_layout_rewrite * [cache flush] port cache flush to ansor (neo-ai#32) * Improve relay integration (neo-ai#34) * tmp checkpoint * Improve relay integration * Improve relay integration * Fix xgb error & Simplify dispatcher (neo-ai#35) * Rename "MetaTileRewritePolicy" to "SketchPolicy". (neo-ai#36) * Rename "MetaTileRewritePolicy" to "SketchPolicy". * Add a new class for auto_unroll_max_step, storage_offset in StageNode * fix tune_op_subgraph.py * rebase * Migrate all node::make to noderef's construct function (neo-ai#37) * Start to move xxxnode::make to noderef() * Update * Update * Finish transform_step * Finish comute dag & auto schedule * Update * Update * Update * Update * Update * Code refine * Code refine * Code refine * Update * Update * Some lint fix & Recover the double constructor of tvm::PrimExpr (neo-ai#39) * lint fix * clang-format-fix * pylint fix * Update * Recover the double constructor of tvm::PrimExpr * Fix pylint * pylint fix * pylint fix * Add MutateComputeLocation and MutateParallel in evolutionary search (neo-ai#40) * Add MutateComputeLocation and MutateParallel in evolutionary search * fix lint * Improve loop state python API (stage_tensors -> stage_ops) (neo-ai#41) * improve loop state python API (stage_tensors -> stage_ops) * fix * ComputeDAG bug fix & Add Custom TensorCore Matmul Example (neo-ai#42) * Bug Fix * Sample example of Custom TensorCore Matmul * Rever Commits, Start to build minimum Ansor system * Code clean for minimum Ansor system * Bug fix & Delete AccessAnalyzer * Delete attachmap & Code clean * Doc update Update statenode::stages from vector to Array * Headfile update & Python doc update * clang-format fix * pylint fix * Update * Doc update * Update * Bug fix after code merge to the new master * clang-format fix * Update * Update * Update std::vector to Array; Update verbosity setting; Some commemts addressed * std::vector->Array & std::string->String * Add init_state to ComputeDAG * Update * Update some unordered_map to Map * clang-format fix * Comments addressed Delete ReplayAndInferBound Delete ReplaySteps & InferBoundCommon * Lint fix * Update * Update * Update * Update * Update * Update * Update * Update * Update * Rename ansor namespace to auto_schedule * Update * Rename ThreadPool to ParallelFor * Add parallel_for * Remove ThreadPool * Update python/tvm/auto_schedule/auto_schedule.py * trigger CI Co-authored-by: Lianmin Zheng <[email protected]> Co-authored-by: Minmin Sun (孙敏敏) <[email protected]> Co-authored-by: Zhao Wu <[email protected]>

…generating (apache#5962) * Code migration Start (#1) * Init commit: Code migration Start * Add loop_state.cc/h * Add ComputeDAG basic test * Split transform_step out & Update more UTs (#3) * Split transform_step out * Update GetProducers & GetConsumers * Update UTs * Add UT for CacheReadWrite & Some bug fix * Add search_task, measure and serialization (#4) * Add FollowSplit & FollowFusedSplit tests * Update dag.InferBound & its UT * Add search_task, measure and serialization * Update Serialization UT * Add MetaTileRewritePolicy (#5) * Add feature * Add cost_model, meta_tile_rewrite_policy * Add MetaTileRewritePolicy basic UT * Basic Python API for State (#6) * Add Basic Python API for State * Add UTs for State * Add Python API: Measure & Task (#7) * Update the return value of state operation * Add task * Copy measure.py & utils.py * Fix LocalBuilder * Fix LocalRunner * Add ansor.auto_schedule() API; First AutoSchedule working version(#8) * Add basic Python support for ansor.auto_schedule * Update AutoSchedule API * Bug fix for get the attach point of a fused iter * Update UT after infer bug fix * Bug fix & Add python serialization API (#10) * Delete C++ UT hack since Python is ready * Add ndarray.non_empty * Update Serialization python API * Improve code style, python wrapper and test cases (#11) * Update c++ code style and unit test * Update python State wrapper and test cases * fix unit tests * Add RPCRunner & OpenCL/CUDA test (#12) * Add RPCRunner & OpenCL search test * Add CUDA search test * Add RPCRunner test * rebase to upstream/master * Add Ansor basic tutorial (#13) * Add basic tutorial * migrate feature extraction (#14) * Add XGBModel & RPCRunnerWarpper (#15) * Add XGBModel & RPCRunnerWarpper * Revert "Add Parallel Granularity Mutation" * Migrate workload_registry.py (#16) * add workload registry * update * update * add task scheduler (#17) * Add conv2d cuda tutorial with workload registry (#18) * add tune_test.py (the old tune_wkl.py) (#19) * add tune_test.py (the old tune_wkl.py) * update * fix measure * fix for gpu * Code refine for tune_test.py & Add a pre load callback (#20) * Bug fix for tutorials * Add PreLoadMeasuredStates * Add search_callback support for task tuner * Code refine for tune_test.py * Update * Update * Update * Update * Bug fix * Add python custom sketch rule (#21) * Add custom sketch rule * Bug fix * Ansor Relay Integration (without layout rewrite) (#22) * relay integration * Add tune_op_subgraph.py & Some code clean for tune_network.py (#23) * Add single op tune scripts * Add tune subgraph support * Merge all op & all subgraph to one file * Rename file * add explicit_unroll_max_extent (#25) * Add Index simplification & API update (#26) * Add vectorized cooperative_fetching test * Update math simplify for vectorized CF * File rename * Update tune_network * API update * Update PreLoadMeasuredStates & Some bug fix (#27) * Add a threading wrapper to fix the test bug * Set default TVM_USE_AUTO_SCHEDULER to false * Update PreLoadMeasuredStates callback * Add tensorize step for loop_state (#31) * Add tensorize step * State python api update (#33) * Start to update api * Add compute_dag to state * API update * kernel layout rewrite (#28) * kernel layout rewrite * remove some hacks * add defuse_ops pass and move kernel_layout_rewrite pass after fuse_ops pass * set TVM_RELAY_DISABLE_BUILD_CACHE for task extraction and prepare_layout_rewrite * [cache flush] port cache flush to ansor (#32) * Improve relay integration (#34) * tmp checkpoint * Improve relay integration * Improve relay integration * Fix xgb error & Simplify dispatcher (#35) * Rename "MetaTileRewritePolicy" to "SketchPolicy". (#36) * Rename "MetaTileRewritePolicy" to "SketchPolicy". * Add a new class for auto_unroll_max_step, storage_offset in StageNode * fix tune_op_subgraph.py * rebase * Migrate all node::make to noderef's construct function (#37) * Start to move xxxnode::make to noderef() * Update * Update * Finish transform_step * Finish comute dag & auto schedule * Update * Update * Update * Update * Update * Code refine * Code refine * Code refine * Update * Update * Some lint fix & Recover the double constructor of tvm::PrimExpr (#39) * lint fix * clang-format-fix * pylint fix * Update * Recover the double constructor of tvm::PrimExpr * Fix pylint * pylint fix * pylint fix * Add MutateComputeLocation and MutateParallel in evolutionary search (#40) * Add MutateComputeLocation and MutateParallel in evolutionary search * fix lint * Improve loop state python API (stage_tensors -> stage_ops) (#41) * improve loop state python API (stage_tensors -> stage_ops) * fix * ComputeDAG bug fix & Add Custom TensorCore Matmul Example (#42) * Bug Fix * Sample example of Custom TensorCore Matmul * Rever Commits, Start to build minimum Ansor system * Code clean for minimum Ansor system * Bug fix & Delete AccessAnalyzer * Delete attachmap & Code clean * Doc update Update statenode::stages from vector to Array * Headfile update & Python doc update * clang-format fix * pylint fix * Update * Doc update * Update * Bug fix after code merge to the new master * clang-format fix * Update * Update * Update std::vector to Array; Update verbosity setting; Some commemts addressed * std::vector->Array & std::string->String * Add init_state to ComputeDAG * Update * Update some unordered_map to Map * clang-format fix * Comments addressed Delete ReplayAndInferBound Delete ReplaySteps & InferBoundCommon * Lint fix * Update * Update * Update * Update * Update * Update * Update * Update * Update * Rename ansor namespace to auto_schedule * Update * Rename ThreadPool to ParallelFor * Add parallel_for * Remove ThreadPool * Update python/tvm/auto_schedule/auto_schedule.py * trigger CI Co-authored-by: Lianmin Zheng <[email protected]> Co-authored-by: Minmin Sun (孙敏敏) <[email protected]> Co-authored-by: Zhao Wu <[email protected]>

trevor-m force-pushed the trevmorr-fix-trt branch 2 times, most recently from 782907d to 71528ba Compare August 28, 2019 22:28

reminisce reviewed Aug 29, 2019

View reviewed changes

trevor-m force-pushed the trevmorr-fix-trt branch 3 times, most recently from 318d071 to ad6f739 Compare October 15, 2019 17:19

yongwww approved these changes Oct 15, 2019

View reviewed changes

trevor-m force-pushed the trevmorr-fix-trt branch from ad6f739 to 6813ac7 Compare December 13, 2019 21:31

zxy844288792 merged commit cfde295 into neo-ai:dev Dec 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix TensorRT integration and add TRT tests to CI #41

Fix TensorRT integration and add TRT tests to CI #41

trevor-m commented Aug 28, 2019 •

edited

Loading

trevor-m commented Aug 29, 2019

reminisce Aug 29, 2019

trevor-m Aug 30, 2019 •

edited

Loading

trevor-m Oct 2, 2019

trevor-m commented Oct 15, 2019

yongwww commented Oct 15, 2019

zhiics commented Oct 15, 2019

trevor-m commented Oct 15, 2019

yongwww left a comment

trevor-m commented Dec 17, 2019

Fix TensorRT integration and add TRT tests to CI #41

Fix TensorRT integration and add TRT tests to CI #41

Conversation

trevor-m commented Aug 28, 2019 • edited Loading

trevor-m commented Aug 29, 2019

reminisce Aug 29, 2019

Choose a reason for hiding this comment

trevor-m Aug 30, 2019 • edited Loading

Choose a reason for hiding this comment

trevor-m Oct 2, 2019

Choose a reason for hiding this comment

trevor-m commented Oct 15, 2019

yongwww commented Oct 15, 2019

zhiics commented Oct 15, 2019

trevor-m commented Oct 15, 2019

yongwww left a comment

Choose a reason for hiding this comment

trevor-m commented Dec 17, 2019

trevor-m commented Aug 28, 2019 •

edited

Loading

trevor-m Aug 30, 2019 •

edited

Loading