[BYOC] Fix build with TensorRT 8

[masahi]

I tried to use TRT BYOC with the latest TensorRT version, but got a build error. The code is using functions that were deprecated in TRT 7 and removed in 8.

These function calls should be redundant, since we are also using config_->setFlag(...) API anyway.

tvm/src/runtime/contrib/tensorrt/tensorrt_builder.cc

Lines 171 to 172 in 6f5b674

	config_->setFlag(nvinfer1::BuilderFlag::kINT8);
	config_->setInt8Calibrator(calibrator_);

@trevor-m @comaniac @tiandiao123

[masahi]

Got MyPy lint errors on files that I didn't touch https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-9047/1/pipeline

[honsontran]

@masahi Question on this. I noticed you deleted the deprecating changes from tensorrt_builder.cc... are you still able to utilize fp16 and int8 functionality for TRT 8 if you removed those flags that were once previously set?

[tiandiao123]

@masahi Question on this. I noticed you deleted the deprecating changes from tensorrt_builder.cc... are you still able to utilize fp16 and int8 functionality for TRT 8 if you removed those flags that were once previously set?

As long as config_ has set up fp16 or int8 when building engine, we don't need to builder_->setFp16Mode(true) or builder_->setInt16Mode(true), and these two functions are deleted in trt8 api.

[tiandiao123]

@masahi Question on this. I noticed you deleted the deprecating changes from tensorrt_builder.cc... are you still able to utilize fp16 and int8 functionality for TRT 8 if you removed those flags that were once previously set?

As long as config_ has set up fp16 or int8 when building engine, we don't need to builder_->setFp16Mode(true) or builder_->setInt16Mode(true), and these two functions are deleted in trt8 api.

I think this pr should be no problem @Laurawly @trevor-m @comaniac

[comaniac]

LGTM

[comaniac]

btw, it would be good if you could also update the TensorRT document (https://tvm.apache.org/docs/deploy/tensorrt.html) about the supported versions.

[honsontran]

Has anyone seen any performance increases with TRT 8?

[tiandiao123]

Has anyone seen any performance increases with TRT 8?

I tested several models such as resnet and super resolution model, and my machine is T4, and they slightly have some performance gain based on my test (probably TRT8 doesn't have too much optimization for vision model? They say they have great performance gain in bert model). However, I didn't test them using large models such as bert, not quite sure whether they have great performance gain in those models, and Nvidia team says bert models have huge performance gain using TRT8. Also, we need to investigate tensorrt 8 supported ops to see what kind of new ops TRT8 has newly-supported, and we can register them into https://github.com/apache/tvm/blob/main/src/runtime/contrib/tensorrt/tensorrt_ops.cc.