Skip to content

Latest commit

 

History

History
 
 

BERT

BERT Inference Using TensorRT

This subfolder of the BERT TensorFlow repository, tested and maintained by NVIDIA, provides scripts to perform high-performance inference using NVIDIA TensorRT.

Table Of Contents

Model overview

BERT, or Bidirectional Encoder Representations from Transformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. This model is based on the BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding paper. NVIDIA's BERT is an optimized version of Google's official implementation, leveraging mixed precision arithmetic and Tensor Cores for faster inference times while maintaining target accuracy.

Other publicly available implementations of BERT include:

  1. NVIDIA PyTorch
  2. Hugging Face
  3. codertimo
  4. gluon-nlp
  5. Google's official implementation

Model architecture

BERT's model architecture is a multi-layer bidirectional Transformer encoder. Based on the model size, we have the following two default configurations of BERT:

Model Hidden layers Hidden unit size Attention heads Feed-forward filter size Max sequence length Parameters
BERT-Base 12 encoder 768 12 4 x 768 512 110M
BERT-Large 24 encoder 1024 16 4 x 1024 512 330M

Typically, the language model is followed by a few task-specific layers. The model used here includes layers for question answering.

TensorRT Inference Pipeline

BERT inference consists of three main stages: tokenization, the BERT model, and finally a projection of the tokenized prediction onto the original text. Since the tokenizer and projection of the final predictions are not nearly as compute-heavy as the model itself, we run them on the host. The BERT model is GPU-accelerated via TensorRT.

The tokenizer splits the input text into tokens that can be consumed by the model. For details on this process, see this tutorial.

To run the BERT model in TensorRT, we construct the model using TensorRT APIs and import the weights from a pre-trained TensorFlow checkpoint from NGC. Finally, a TensorRT engine is generated and serialized to the disk. The various inference scripts then load this engine for inference.

Lastly, the tokens predicted by the model are projected back to the original text to get a final result.

Version Info

The following software version configuration has been tested:

Software Version
Python >=3.6.x
TensorRT 8.0.1.6
CUDA 11.3.1

Setup

The following section lists the requirements that you need to meet in order to run the BERT model.

Requirements

This demo BERT application can be run within the TensorRT OSS build container. If running in a different environment, following packages are required.

Quick Start Guide

  1. Build and launch the container as described in TensorRT OSS README.

    Note: After this point, all commands should be run from within the container.

  2. Verify TensorRT installation by printing the version: For example:

    python3 -c "import tensorrt as trt; print(trt.__version__)"
  3. Download the SQuAD dataset and BERT checkpoints:

    cd $TRT_OSSPATH/demo/BERT

    Download SQuAD v1.1 training and dev dataset.

    bash ./scripts/download_squad.sh

    Download Tensorflow checkpoints for BERT large model with sequence length 128, fine-tuned for SQuAD v2.0.

    bash scripts/download_model.sh

Note: Since the datasets and checkpoints are stored in the directory mounted from the host, they do not need to be downloaded each time the container is launched.

  1. Build a TensorRT engine. To build an engine, run the builder.py script. For example:

    mkdir -p engines && python3 builder.py -m models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/model.ckpt -o engines/bert_large_128.engine -b 1 -s 128 --fp16 -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1

    This will build an engine with a maximum batch size of 1 (-b 1), and sequence length of 128 (-s 128) using mixed precision (--fp16) using the BERT Large SQuAD v2 FP16 Sequence Length 128 checkpoint (-c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1).

  2. Run inference. Two options are provided for running the model.

    a. inference.py script This script accepts a passage and question and then runs the engine to generate an answer. For example:

    python3 inference.py -e engines/bert_large_128.engine -p "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps." -q "What is TensorRT?" -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/vocab.txt

    b. inference.ipynb Jupyter Notebook The Jupyter Notebook includes a passage and various example questions and allows you to interactively make modifications and see the outcome. To launch the Jupyter Notebook from inside the container, run:

    jupyter notebook --ip 0.0.0.0 inference.ipynb

    Then, use your browser to open the link displayed. The link should look similar to: http://127.0.0.1:8888/?token=<TOKEN>

  3. Run inference with CUDA Graph support.

    A separate python inference_c.py script is provided to run inference with CUDA Graph support. This is necessary since CUDA Graph is only supported through CUDA C/C++ APIs, not pyCUDA. The inference_c.py script uses pybind11 to interface with C/C++ for CUDA graph capturing and launching. The cmdline interface is the same as inference.py except for an extra --enable-graph option.

    mkdir -p build; pushd build
    cmake .. -DPYTHON_EXECUTABLE=$(which python)
    make -j
    popd
    python3 inference_c.py -e engines/bert_large_128.engine --enable-graph -p "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps." -q "What is TensorRT?" -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/vocab.txt

    A separate C/C++ inference benchmark executable perf (compiled from perf.cpp) is provided to run inference benchmarks with CUDA Graph. The cmdline interface is the same as perf.py except for an extra --enable_graph option.

    build/perf -e engines/bert_large_128.engine -b 1 -s 128 -w 100 -i 1000 --enable_graph

(Optional) Trying a different configuration

If you would like to run another configuration, you can manually download checkpoints using the included script. For example, run:

bash scripts/download_model.sh base

to download a BERT Base model instead of the default BERT Large model.

To view all available model options, run:

bash scripts/download_model.sh -h

Advanced

The following sections provide greater details on inference with TensorRT.

Scripts and sample code

In the root directory, the most important files are:

  • builder.py - Builds an engine for the specified BERT model
  • Dockerfile - Container which includes dependencies and model checkpoints to run BERT
  • inference.ipynb - Runs inference interactively
  • inference.py - Runs inference with a given passage and question
  • perf.py - Runs inference benchmarks

The scripts/ folder encapsulates all the one-click scripts required for running various supported functionalities, such as:

  • build.sh - Builds a Docker container that is ready to run BERT
  • launch.sh - Launches the container created by the build.sh script.
  • download_model.sh - Downloads pre-trained model checkpoints from NGC
  • inference_benchmark.sh - Runs an inference benchmark and prints results

Other folders included in the root directory are:

  • helpers - Contains helpers for tokenization of inputs

The infer_c/ folder contains all the necessary C/C++ files required for CUDA Graph support.

  • bert_infer.h - Defines necessary data structures for running BERT inference
  • infer_c.cpp - Defines C/C++ interface using pybind11 that can be plugged into inference_c.py
  • perf.cpp - Runs inference benchmarks. It is equivalent to perf.py, with an extra option --enable_graph to enable CUDA Graph support.

Command-line options

To view the available parameters for each script, you can use the help flag (-h).

TensorRT inference process

As mentioned in the Quick Start Guide, two options are provided for running inference:

  1. The inference.py script which accepts a passage and a question and then runs the engine to generate an answer. Alternatively, this script can be used to run inference on the Squad dataset.
  2. The inference.ipynb Jupyter Notebook which includes a passage and various example questions and allows you to interactively make modifications and see the outcome.

Accuracy

Evaluating PTQ (post-training quantization) Int8 Accuracy Using The SQuAD Dataset

  1. Download Tensorflow checkpoints for a BERT Large FP16 SQuAD v2 model with a sequence length of 384:

    bash scripts/download_model.sh large 384 v2
  2. Build an engine:

    Turing and Ampere GPUs

    # QKVToContextPlugin and SkipLayerNormPlugin supported with INT8 I/O. To enable, use -imh and -iln builder flags respectively.
    mkdir -p engines && python3 builder.py -m models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/model.ckpt -o engines/bert_large_384_int8mix.engine -b 1 -s 384 --int8 --fp16 --strict -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 --squad-json ./squad/train-v1.1.json -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt --calib-num 100 -iln -imh

    Xavier GPU

    # Only supports SkipLayerNormPlugin running with INT8 I/O. Use -iln builder flag to enable.
    mkdir -p engines && python3 builder.py -m models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/model.ckpt -o engines/bert_large_384_int8mix.engine -b 1 -s 384 --int8 --fp16 --strict -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 --squad-json ./squad/train-v1.1.json -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt --calib-num 100 -iln 

    Volta GPU

    # No support for QKVToContextPlugin or SkipLayerNormPlugin running with INT8 I/O. Don't specify -imh or -iln in builder flags.
    mkdir -p engines && python3 builder.py -m models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/model.ckpt -o engines/bert_large_384_int8mix.engine -b 1 -s 384 --int8 --fp16 --strict -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 --squad-json ./squad/train-v1.1.json -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt --calib-num 100

    This will build an engine with a maximum batch size of 1 (-b 1), calibration dataset squad (--squad-json ./squad/train-v1.1.json), calibration sentences number 100 (--calib-num 100), and sequence length of 384 (-s 384) using INT8 mixed precision computation where possible (--int8 --fp16 --strict).

  3. Run inference using the squad dataset, and evaluate the F1 score and exact match score:

    python3 inference.py -e engines/bert_large_384_int8mix.engine -s 384 -sq ./squad/dev-v1.1.json -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -o ./predictions.json
    python3 squad/evaluate-v1.1.py  squad/dev-v1.1.json  ./predictions.json 90

Evaluating QAT (quantization aware training) Int8 Accuracy Using The SQuAD Dataset

  1. Download checkpoint for BERT Large FP16 SQuAD v1.1 model with sequence length of 384:

    bash scripts/download_model.sh pyt v1_1
  2. Build an engine:

    Turing and Ampere GPUs

    # QKVToContextPlugin and SkipLayerNormPlugin supported with INT8 I/O. To enable, use -imh and -iln builder flags respectively.
    mkdir -p engines && python3 builder.py -o engines/bert_large_384_int8mix.engine -b 1 -s 384 --int8 --fp16 --strict -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -x models/fine-tuned/bert_pyt_onnx_large_qa_squad11_amp_fake_quant_v1/bert_large_v1_1_fake_quant.onnx -iln -imh

    Xavier GPU

    # Only supports SkipLayerNormPlugin running with INT8 I/O. Use -iln builder flag to enable.
    mkdir -p engines && python3 builder.py -o engines/bert_large_384_int8mix.engine -b 1 -s 384 --int8 --fp16 --strict -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -x models/fine-tuned/bert_pyt_onnx_large_qa_squad11_amp_fake_quant_v1/bert_large_v1_1_fake_quant.onnx -iln 

    Volta GPU

    # No support for QKVToContextPlugin or SkipLayerNormPlugin running with INT8 I/O. Don't specify -imh or -iln in builder flags.
    mkdir -p engines && python3 builder.py -o engines/bert_large_384_int8mix.engine -b 1 -s 384 --int8 --fp16 --strict -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -x models/fine-tuned/bert_pyt_onnx_large_qa_squad11_amp_fake_quant_v1/bert_large_v1_1_fake_quant.onnx 

    This will build and engine with a maximum batch size of 1 (-b 1) and sequence length of 384 (-s 384) using INT8 mixed precision computation where possible (--int8 --fp16 --strict).

  3. Run inference using the squad dataset, and evaluate the F1 score and exact match score:

    python3 inference.py -e engines/bert_large_384_int8mix.engine -s 384 -sq ./squad/dev-v1.1.json -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -o ./predictions.json
    python3 squad/evaluate-v1.1.py  squad/dev-v1.1.json  ./predictions.json 90

Experimental

Variable sequence length

In our prior implementation, we used inputs padded to max length along with corresponding input masks to handle variable sequence length inputs in a batch. The padding results in some wasted computations which can be avoided by handling variable sequence length inputs natively. Now we have a new approach called the variable sequence length method. By concatenating each input id into a single long input id, and concatenating each input segment id into a single long segment id, TensorRT can know the exact starts and ends by providing an extra sequence length buffer that contains the start and end positions of each sequence. Now we can eliminate the wasted computation in the input paddings.

Note this is an experimental feature because we only support Xavier+ GPUs, also there is neither FP32 support nor INT8 PTQ calibration.

  1. Download checkpoint for BERT Large FP16 SQuAD v1.1 model with sequence length of 384:

    bash scripts/download_model.sh pyt v1_1
  2. Build an engine:

    FP16 engine

    mkdir -p engines && python3 builder_varseqlen.py -x models/fine-tuned/bert_pyt_onnx_large_qa_squad11_amp_fake_quant_v1/bert_large_v1_1_fake_quant.onnx -o engines/bert_varseq_fp16.engine -b 1 -s 64 --fp16 -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt

    This will build and engine with a maximum batch size of 1 (-b 1) and sequence length of 64 (-s 64) using FP16 precision computation where possible (--fp16).

    INT8 engine

    mkdir -p engines && python3 builder_varseqlen.py -x models/fine-tuned/bert_pyt_onnx_large_qa_squad11_amp_fake_quant_v1/bert_large_v1_1_fake_quant.onnx -o engines/bert_varseq_int8.engine -b 1 -s 256 --int8 --fp16 -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt

    This will build and engine with a maximum batch size of 1 (-b 1) and sequence length of 256 (-s 256) using INT8 precision computation where possible (--int8).

  3. Run inference

    Evaluate the F1 score and exact match score using the squad dataset:

    python3 inference_varseqlen.py -e engines/bert_varseq_int8.engine -s 256 -sq ./squad/dev-v1.1.json -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -o ./predictions.json
    python3 squad/evaluate-v1.1.py  squad/dev-v1.1.json  ./predictions.json 90

    Run the quesion and answer mode:

    python3 inference_varseqlen.py -e engines/bert_varseq_int8.engine -p "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps." -q "What is TensorRT?" -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -s 256
  4. Collect performance data

    python3 perf_varseqlen.py -e engines/bert_varseq_int8.engine -b 1 -s 256

    This will collect performance data run use batch size 1 (-b 1) and sequence length of 256 (-s 256).

  5. Collect performance data with CUDA graph enabled

    We can use the same inference_c.py and build/perf to collect performance data with cuda graph enabled. The command line is the same as run without variable sequence length.

Sparsity with Quantization Aware Training

Fine-grained 2:4 structured sparsity support introduced in NVIDIA Ampere GPUs can produce significant performance gains in BERT inference. The network is first trained using dense weights, then fine-grained structured pruning is applied, and finally the remaining non-zero weights are fine-tuned with additional training steps. This method results in virtually no loss in inferencing accuracy.

Using INT8 precision with quantization scales obtained from Post-Training Quantization (PTQ) can produce additional performance gains, but may also result in accuracy loss. Alternatively, for PyTorch-trained models, NVIDIA PyTorch-Quantization toolkit can be leveraged to perform quantized fine tuning (a.k.a. Quantization Aware Training or QAT) and generate the INT8 quantization scales as part of training. This generally results in higher accuracy compared to PTQ.

To demonstrate the potential speedups from these optimizations in demoBERT, we provide the Megatron-LM transformer model finetuned for SQuAD 2.0 task with sparsity and quantization.

The sparse weights are generated by finetuning with INT8 Quantization Aware Training recipe. This feature can be used with the fixed or variable sequence length implementations by passing in -sp flag to demoBERT builder.

Megatron-LM for Question Answering

Example: Megatron-LM Large SQuAD v2.0 with sparse weights for sequence length 384

Build the TensorRT engine:

Options specified:

  • --megatron : assume Megatron style residuals instead of vanilla BERT.
  • --pickle : specify a pickle file containing the PyTorch statedict corresponding to fine-tuned Megatron model.
  • -sp : enable sparsity during engine optimization and treat the weights as sparse.
  • --int8 --il : enable int8 tactics/plugins with interleaving.
bash ./scripts/download_model.sh 384 v1_1 # BERT-large model checkpoint fine-tuned for SQuAD 1.1
bash ./scripts/download_model.sh pyt megatron-large int8-qat sparse # Megatron-LM model weights
export CKPT_PATH=models/fine-tuned/bert_pyt_statedict_megatron_sparse_int8qat_v21.03.0/bert_pyt_statedict_megatron_sparse_int8_qat
mkdir -p engines && python3 builder_varseqlen.py -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -b 1 -s 384 -o engines/megatron_large_seqlen384_int8qat_sparse.engine --fp16 --int8 --strict -il --megatron --pickle $CKPT_PATH -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -sp

Ask a question:

python3 inference_varseqlen.py -e engines/megatron_large_seqlen384_int8qat_sparse.engine -p "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps." -q "What is TensorRT?" -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -s 256

Evaluate F1 score:

python3 inference_varseqlen.py -e engines/megatron_large_seqlen384_int8qat_sparse.engine -s 384 -sq ./squad/dev-v1.1.json -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -o ./predictions.json
python3 squad/evaluate-v1.1.py  squad/dev-v1.1.json  ./predictions.json 90

Expected output:

&&&& PASSED TensorRT BERT Squad Accuracy matches reference.
{"exact_match": 84.03973509933775, "f1": 90.88667129897755}

Performance

Benchmarking

The following section shows how to run the inference benchmarks for BERT.

TensorRT inference benchmark

The inference benchmark is performed on a single GPU by the inference_benchmark.sh script, which takes the following steps for each set of model parameters:

  1. Downloads checkpoints and builds a TensorRT engine if it does not already exist.

  2. Runs 100 warm-up iteration then runs inference for 1000 to 2000 iterations for each batch size specified in the script, selecting the profile best for each size.

Note: The time measurements do not include the time required to copy inputs to the device and copy outputs to the host.

To run the inference benchmark script, run:

bash scripts/inference_benchmark.sh --gpu <arch>

Options for <arch> are: 'Volta', 'Xavier', 'Turing', 'Ampere'

Note: Some of the configurations in the benchmark script require 16GB of GPU memory. On GPUs with smaller amounts of memory, parts of the benchmark may fail to run.

Also note that BERT Large engines, especially using mixed precision with large batch sizes and sequence lengths may take a couple hours to build.

Results

The following sections provide details on how we achieved our performance and inference.

Inference performance: NVIDIA A100 (40GB)

Our results for BERT were obtained by running the scripts/inference_benchmark.sh --gpu Ampere script in the container generated by the TensorRT OSS Dockerfile on NVIDIA A100 with (1x A100 40G) GPUs.

BERT Base
Sequence Length Batch Size INT8 Latency (ms) FP16 Latency (ms)
95th Percentile 99th Percentile Average 95th Percentile 99th Percentile Average
128 1 0.33 0.97 0.58 0.75 0.75 0.72
128 2 0.78 0.79 0.63 0.84 1.07 0.84
128 4 0.76 0.98 0.76 1.13 1.46 1.14
128 8 1.08 1.08 0.98 1.66 1.81 1.66
128 12 1.26 1.63 1.27 2.07 2.07 2.07
128 16 1.47 1.48 1.47 2.48 2.49 2.48
128 24 2.13 2.13 2.13 3.47 3.49 3.46
128 32 2.54 2.83 2.54 4.37 4.40 4.34
128 64 4.58 4.59 4.54 8.70 8.79 8.65
128 128 9.04 9.06 8.97 17.05 17.07 16.90
384 1 1.15 1.15 1.15 1.43 1.44 1.43
384 2 1.37 1.37 1.37 1.84 2.21 1.84
384 4 1.73 1.74 1.73 2.47 2.48 2.47
384 8 2.51 2.51 2.51 3.77 3.80 3.76
384 12 3.61 3.62 3.61 5.36 5.37 5.30
384 16 4.39 4.40 4.38 7.32 7.32 7.24
384 24 6.24 6.24 6.23 10.50 10.51 10.41
384 32 8.42 8.50 8.42 14.32 14.44 14.27
384 64 16.48 16.52 16.36 27.51 27.54 27.33
384 128 31.71 31.78 31.58
BERT Large
Sequence Length Batch Size INT8 Latency (ms) FP16 Latency (ms)
95th Percentile 99th Percentile Average 95th Percentile 99th Percentile Average
128 1 1.24 1.56 1.24 1.73 2.11 1.73
128 2 1.49 1.49 1.49 2.20 2.20 2.20
128 4 1.91 1.92 1.91 3.22 3.23 3.22
128 8 2.94 2.94 2.93 4.84 4.84 4.83
128 12 3.34 3.34 3.34 5.95 5.96 5.90
128 16 4.63 4.64 4.62 7.98 7.99 7.90
128 24 5.87 5.88 5.87 11.05 11.08 10.94
128 32 7.99 7.99 7.98 14.74 14.77 14.59
128 64 14.74 17.74 14.56 28.09 28.25 27.85
128 128 28.32 23.38 28.03 54.38 54.40 54.12
384 1 2.80 2.80 2.80 3.49 3.49 3.48
384 2 3.12 3.13 3.12 4.71 4.72 4.71
384 4 4.27 4.27 4.27 6.70 6.71 6.70
384 8 7.66 7.67 7.66 12.41 12.53 12.37
384 12 10.07 10.08 10.07 17.63 17.76 17.56
384 16 13.34 13.34 13.33 23.40 23.46 23.19
384 24 19.36 19.38 19.22 34.34 34.36 34.10
384 32 25.56 25.60 25.56 44.94 44.98 44.78
384 64 49.84 49.92 49.60 87.26 87.56 86.77
384 128 97.66 97.78 97.06 170.85 171.00 170.08
Megatron Large with Sparsity
Sequence Length Batch Size INT8 QAT Latency (ms)
95th Percentile 99th Percentile Average
128 1 1.16 1.46 1.17
128 2 1.44 1.45 1.44
128 4 1.69 1.7 1.69
128 8 2.34 2.34 2.34
128 12 2.8 2.8 2.8
128 16 3.7 3.71 3.7
128 24 4.63 4.63 4.62
128 32 6.33 6.33 6.32
128 64 11.34 11.35 11.24
128 128 21.18 21.19 21.06
384 1 1.61 1.61 1.61
384 2 2.19 2.19 2.18
384 4 3.31 3.31 3.31
384 8 5.48 5.48 5.47
384 12 7.69 7.7 7.69
384 16 10.02 10.02 10.01
384 24 14.15 14.15 14.14
384 32 18.41 18.56 18.4
384 64 35.71 35.73 35.44
384 128 68.52 68.55 68.19

Inference performance: NVIDIA T4 (16GB)

Our results were obtained by running the scripts/inference_benchmark.sh --gpu Turing script in the container generated by the TensorRT OSS Dockerfile on NVIDIA T4 with (1x T4 16G) GPUs.

BERT Base
Sequence Length Batch Size INT8 Latency (ms) FP16 Latency (ms)
95th Percentile 99th Percentile Average 95th Percentile 99th Percentile Average
128 1 1.55 1.57 1.33 2.00 2.06 1.93
128 2 1.78 2.06 1.75 2.54 2.58 2.49
128 4 2.80 2.88 2.74 4.25 4.34 4.16
128 8 4.48 4.56 4.42 8.13 8.74 7.88
128 12 6.28 6.31 6.12 11.67 12.12 11.30
128 16 8.92 9.11 8.78 17.24 17.79 16.70
128 24 12.70 12.84 12.53 24.48 24.85 24.90
128 32 17.90 18.41 17.59 33.02 33.51 32.65
128 64 34.80 34.83 34.31 65.38 65.43 64.28
128 128 68.16 68.46 67.05 130.77 131.01 129.19
384 1 2.47 2.53 2.43 3.76 3.81 3.69
384 2 3.87 3.95 3.81 6.31 6.43 6.21
384 4 7.15 7.18 6.97 12.16 12.22 12.03
384 8 14.09 12.11 13.73 25.45 25.83 24.94
384 12 20.99 21.12 20.66 38.15 38.38 37.51
384 16 27.49 27.65 27.08 50.90 51.36 50.04
384 24 41.93 42.17 41.36 77.25 78.16 76.05
384 32 54.65 54.87 54.06 102.44 103.09 101.30
384 64 109.78 110.42 108.24 200.58 201.20 198.68
384 128 227.46 228.80 223.92 401.33 402.14 399.24
BERT Large
Sequence Length Batch Size INT8 Latency (ms) FP16 Latency (ms)
95th Percentile 99th Percentile Average 95th Percentile 99th Percentile Average
128 1 3.59 3.61 3.51 5.10 5.18 5.02
128 2 4.93 5.03 4.83 7.72 7.73 7.58
128 4 8.15 8.19 7.93 13.67 13.85 13.56
128 8 14.21 14.23 13.89 26.88 27.66 26.35
128 12 22.41 22.47 21.91 41.04 41.29 40.30
128 16 29.30 29.83 28.82 55.04 55.27 54.05
128 24 44.60 44.63 43.92 81.24 82.28 79.59
128 32 60.88 61.48 58.97 114.13 114.47 112.78
128 64 111.78 112.02 110.77 224.24 225.02 221.97
128 128 223.99 224.28 222.33 417.56 418.54 415.33
384 1 7.18 7.27 7.07 11.74 11.96 11.51
384 2 12.22 12.25 11.92 21.47 21.61 20.97
384 4 35.95 36.43 35.63 42.03 42.35 41.36
384 8 47.06 47.22 46.41 83.16 83.51 82.06
384 12 66.04 66.04 65.89 127.10 127.99 127.46
384 16 87.98 88.45 87.13 164.13 165.12 161.96
384 24 132.56 132.96 131.24 262.76 263.68 258.96
384 32 179.44 180.61 176.66 329.99 331.67 325.59
384 64 352.81 353.39 350.21 684.19 686.39 674.76
384 128 706.85 707.73 704.38 1318.74 1320.22 1315.10