ptq

Step-by-Step

This document list steps of reproducing Intel Optimized TensorFlow image recognition models tuning results via Neural Compressor.

Note: Most of those models are both supported in Intel optimized TF 1.15.x and Intel optimized TF 2.x. Version support

Prerequisite

1. Installation

Recommend python 3.6 or higher version.

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
pip install -r requirements.txt

2. Prepare Dataset

TensorFlow models repo provides scripts and instructions to download, process and convert the ImageNet dataset to the TF records format. We also prepared related scripts in imagenet_prepare directory. To download the raw images, the user must create an account with image-net.org. If you have downloaded the raw data and preprocessed the validation data by moving the images into the appropriate sub-directory based on the label (synset) of the image. we can use below command ro convert it to tf records format.

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
# convert validation subset
bash prepare_dataset.sh --output_dir=./data --raw_dir=/PATH/TO/img_raw/val/ --subset=validation
# convert train subset
bash prepare_dataset.sh --output_dir=./data --raw_dir=/PATH/TO/img_raw/train/ --subset=train

EfficientNet-b0 model use ImageNet raw and caffe label. Please download caffe label.

cd /PATH/TO/img_raw/val/
wget http://dl.caffe.berkeleyvision.org/caffe_ilsvrc12.tar.gz
tar -xvf caffe_ilsvrc12.tar.gz

3. Prepare pre-trained model

In this version, Intel® Neural Compressor just support PB file as input for TensorFlow backend, so we need prepared model pre-trained pb files. For some models pre-trained pb can be found in IntelAI Models, we can found the download link in README file of each model. And for others models in Google models, we can get the pb files by convert the checkpoint files. We will give a example with Inception_v1 to show how to get the pb file by a checkpoint file.

1. Download the checkpoint file from here

wget http://download.tensorflow.org/models/inception_v1_2016_08_28.tar.gz
tar -xvf inception_v1_2016_08_28.tar.gz

2. Exporting the Inference Graph

git clone https://github.com/tensorflow/models
cd models/research/slim
python export_inference_graph.py \
        --alsologtostderr \
        --model_name=inception_v1 \
        --output_file=/tmp/inception_v1_inf_graph.pb

Make sure to use intel-tensorflow v1.15, and pip install tf_slim.

Install Intel Tensorflow 1.15 up2

Check your python version and use pip install 1.15.0 up2 from links below: https://storage.googleapis.com/intel-optimized-tensorflow/intel_tensorflow-1.15.0up2-cp36-cp36m-manylinux2010_x86_64.whl
https://storage.googleapis.com/intel-optimized-tensorflow/intel_tensorflow-1.15.0up2-cp37-cp37m-manylinux2010_x86_64.whl https://storage.googleapis.com/intel-optimized-tensorflow/intel_tensorflow-1.15.0up2-cp35-cp35m-manylinux2010_x86_64.whl

Please note: The ImageNet dataset has 1001, the VGG and ResNet V1 final layers have only 1000 outputs rather than 1001. So we need add the --labels_offset=1 flag in the inference graph exporting command.

3. Use Netron to get the input/output layer name of inference graph pb, for Inception_v1 the output layer name is InceptionV1/Logits/Predictions/Reshape_1

4. Freezing the exported Graph, please use the tool freeze_graph.py in tensorflow v1.15.2 repo

python freeze_graph.py \
        --input_graph=/tmp/inception_v1_inf_graph.pb \
        --input_checkpoint=./inception_v1.ckpt \
        --input_binary=true \
        --output_graph=./frozen_inception_v1.pb \
        --output_node_names=InceptionV1/Logits/Predictions/Reshape_1

Run

Note: The model name with * means it comes from models, please follow the step Prepare pre-trained model to get the pb files. The densenet-series comes from tensorflow-densenet, please also follow the step Prepare pre-trained model to get the pb files or use openvino download tools.

git clone https://github.com/openvinotoolkit/open_model_zoo.git
cd open_model_zoo/tools/downloader
git checkout tags/2021.2
pip install -r requirements.in
python downloader.py --name densenet-{121|161|169}-tf -o /PATH/TO/MODEL

1. ResNet50 V1.0

Download pre-trained PB

wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/resnet50_fp32_pretrained_model.pb

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=resnet50_v1.yaml \
    --input_model=/PATH/TO/resnet50_fp32_pretrained_model.pb \
    --output_model=./nc_resnet50_v1.pb

2. ResNet50 V1.5

Download pre-trained PB

wget https://zenodo.org/record/2535873/files/resnet50_v1.pb

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=resnet50_v1_5.yaml \
        --input_model=/PATH/TO/resnet50_v1.pb --output_model=./nc_resnet50_v15.pb

3. ResNet101

Download pre-trained PB

wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/resnet101_fp32_pretrained_model.pb

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=resnet101.yaml \
    --input_model=/PATH/TO/resnet101_fp32_pretrained_model.pb \
    --output_model=./nc_resnet101.pb

4. MobileNet V1

Download pre-trained PB

wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/mobilenet_v1_1.0_224_frozen.pb

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=mobilenet_v1.yaml \
    --input_model=/PATH/TO/mobilenet_v1_1.0_224_frozen.pb \
    --output_model=./nc_mobilenetv1.pb

5. MobileNet V2*

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=mobilenet_v2.yaml \
    --input_model=/PATH/TO/frozen_mobilenet_v2.pb \
    --output_model=./nc_mobilenetv2.pb

6. Inception V1*

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=inception_v1.yaml \
    --input_model=/PATH/TO/frozen_inception_v1.pb \
    --output_model=./nc_inceptionv1.pb

7. Inception V2*

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=inception_v2.yaml \
    --input_model=/PATH/TO/frozen_inception_v2.pb \
    --output_model=./nc_inceptionv2.pb

8. Inception V3

Download pre-trained PB

wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/inceptionv3_fp32_pretrained_model.pb

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=inception_v3.yaml \
    --input_model=/PATH/TO/inceptionv3_fp32_pretrained_model.pb \
    --output_model=./nc_inceptionv3.pb

9. Inception V4

Download pre-trained PB

wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/inceptionv4_fp32_pretrained_model.pb

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=inception_v4.yaml \
    --input_model=/PATH/TO/inceptionv4_fp32_pretrained_model.pb \
    --output_model=./nc_inceptionv4.pb

10. Inception ResNet V2*

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=inception_resnet_v2.yaml \
    --input_model=/PATH/TO/frozen_inception_resnet_v2.pb \
    --output_model=./nc_irv2.pb

11. VGG 16*

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=vgg16.yaml \
        --input_model=/PATH/TO/frozen_vgg16.pb --output_model=./nc_vgg16.pb

12. VGG 19*

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=vgg19.yaml \
        --input_model=/PATH/TO/frozen_vgg19.pb --output_model=./nc_vgg19.pb

13. ResNet v2 50

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=resnet_v2_50.yaml \
        --input_model=/PATH/TO/frozen_resnet50v2_50.pb --output_model=./nc_resnetv2_50.pb

14. ResNet v2 101

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=resnet_v2_101.yaml \
        --input_model=/PATH/TO/frozen_resnetv2_101.pb --output_model=./nc_resnetv2_101.pb

15. ResNet v2 152

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=resnet_v2_152.yaml \
    --input_model=/PATH/TO/frozen_resnetv2_152.pb \
    --output_model=./nc_resnetv2_152.pb

16. Densenet-121

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=densenet121.yaml \
        --input_model=/PATH/TO/densenet121.pb --output_model=./nc_densenet121

17. Densenet-161

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=densenet161.yaml \
        --input_model=/PATH/TO/densenet161.pb --output_model=./nc_densenet161

18. Densenet-169

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=densenet169.yaml \
        --input_model=/PATH/TO/densenet169.pb --output_model=./nc_densenet169

19. Nasnet-mobile*

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=nasnet_mobile.yaml \
        --input_model=/PATH/TO/frozen_nasnet_mobile.pb --output_model=./nc_nasnet_mobile

20. EfficientNet-b0

Download pre-trained checkpoint

wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/ckpts/efficientnet-b0.tar.gz
tar -xvf efficientnet-b0.tar.gz

cd examples/tensorflow/image_recognition/tensorflow_models/quantization/ptq
bash run_tuning.sh --config=efficientnet-b0.yaml \
    --input_model=/PATH/TO/efficientnet-b0 \
    --output_model=./nc_efficientnet-b0.pb

Examples of enabling Intel® Neural Compressor auto tuning on TensorFlow ResNet50 V1.5

This is a tutorial of how to enable a TensorFlow image recognition model with Intel® Neural Compressor.

User Code Analysis

Intel® Neural Compressor supports two usages:

1. User specifies fp32 "model", yaml configured calibration dataloader in calibration field and evaluation dataloader in evaluation field, metric in tuning.metric field of model-specific yaml config file.

Note: you should change the model-specific yaml file dataset path to your own dataset path

2. User specifies fp32 "model", calibration dataset "q_dataloader" and a custom "eval_func" which encapsulates the evaluation dataset and metric by itself.

As ResNet50 V1.5 is a typical image recognition model, use Top-K as metric which is built-in supported by Intel® Neural Compressor. So here we integrate Tensorflow ResNet50 V1.5 in IntelAI Models with Intel® Neural Compressor by the first use case for simplicity.

Write Yaml config file

In examples directory, there is a template.yaml. We could remove most of the items and only keep mandatory item for tuning.

# resnet50_v1_5.yaml

model:                                               # mandatory. used to specify model specific information.
  name: resnet50_v1_5
  framework: tensorflow                              # mandatory. supported values are tensorflow, pytorch, pytorch_ipex, onnxrt_integer, onnxrt_qlinear or mxnet; allow new framework backend extension.
  inputs: input_tensor
  outputs: softmax_tensor

quantization:                                        # optional. tuning constraints on model-wise for advance user to reduce tuning space.
  calibration:
    sampling_size: 5, 10                             # optional. default value is 100. used to set how many samples should be used in calibration.
    dataloader:
      dataset:
        ImageRecord:
          root: /path/to/calibration/dataset         # NOTE: modify to calibration dataset location if needed
      transform:
        ResizeCropImagenet: 
          height: 224
          width: 224
          mean_value: [123.68, 116.78, 103.94]
  model_wise:                                        # optional. tuning constraints on model-wise for advance user to reduce tuning space.
    activation:
      algorithm: minmax

evaluation:                                          # optional. required if user doesn't provide eval_func in neural_compressor.Quantization.
  accuracy:                                          # optional. required if user doesn't provide eval_func in neural_compressor.Quantization.
    metric:
      topk: 1                                        # built-in metrics are topk, map, f1, allow user to register new metric.
    dataloader:
      batch_size: 10
      dataset:
        ImageRecord:
          root: /path/to/evaluation/dataset          # NOTE: modify to evaluation dataset location if needed
      transform:
        ResizeCropImagenet: 
          height: 224
          width: 224
          mean_value: [123.68, 116.78, 103.94]
  performance:                                       # optional. used to benchmark performance of passing model.
    configs:
      cores_per_instance: 4
      num_of_instance: 7
    dataloader:
      batch_size: 1 
      dataset:
        ImageRecord:
          root: /path/to/evaluation/dataset          # NOTE: modify to evaluation dataset location if needed
      transform:
        ResizeCropImagenet: 
          height: 224
          width: 224
          mean_value: [123.68, 116.78, 103.94]

tuning:
  accuracy_criterion:
    relative:  0.01                                  # optional. default value is relative, other value is absolute. this example allows relative accuracy loss: 1%.
  exit_policy:
    timeout: 0                                       # optional. tuning timeout (seconds). default value is 0 which means early stop. combine with max_trials field to decide when to exit.
  random_seed: 9527                                  # optional. random seed for deterministic tuning.

Here we choose topk which is built-in metric and set accuracy criterion as tolerating 0.01 relative accuracy loss of baseline. The default tuning strategy is basic strategy. The timeout 0 means early stop as long as a tuning config meet accuracy target.

preparation

There are three preparation steps in here:

1. Prepare environment

pip install intel-tensorflow==1.15.2 neural_compressor

2. Get the model source code

git clone -b v1.6.0 https://github.com/IntelAI/models intelai_models
cd intelai_models/models/image_recognition/tensorflow/resnet50v1_5/inference

3. Prepare the ImageNet dataset and pretrained PB file

wget https://zenodo.org/record/2535873/files/resnet50_v1.pb

code update

After completed preparation steps, we just need to add below tuning part in eval_classifier_optimized_graph class.

  def auto_tune(self):
    """This is Intel® Neural Compressor tuning part to generate a quantized pb

    Returns:
        graph: it will return a quantized pb
    """
    from neural_compressor.experimental import Quantization, common
    quantizer = Quantization(self.args.config)
    quantizer.model = common.Model(self.args.input_graph)
    q_model = quantizer.fit()
    return q_model

Finally, add one line in __main__ function of eval_image_-classifier_inference.py to use Intel® Neural Compressor by yourself as below.

q_graph = evaluate_opt_graph.auto_tune()

The quantizer.fit() function will return a best quantized model within timeout constrain.