Running the TensorFlow Official ResNet with TensorRT

Deepdrive docs
Search…
Deepdrive docs
Deepdrive
vendor
openai
tensorflow
Surgically extracted MobileNetV2 from tensorflow/models @ 84da970ee43c04fbd53a1db3c824ea32cec8936b
TensorFlow Research Models
slim
Running the TensorFlow Official ResNet with TensorRT
Deepdrive
docs
README
Running the TensorFlow Official ResNet with TensorRT
​TensorRT is NVIDIA's inference optimizer for deep learning. Briefly, TensorRT rewrites parts of the execution graph to allow for faster prediction times.
Here we provide a sample script that can:
1.Convert a TensorFlow SavedModel to a Frozen Graph.
2.Load a Frozen Graph for inference.
3.Time inference loops using the native TensorFlow graph.
4.Time inference loops using FP32, FP16, or INT8 precision modes from TensorRT.
We provide some results below, as well as instructions for running this script.
How to Run This Script
Step 1: Install Prerequisites
1.​Install TensorFlow.​
2.​Install TensorRT.​
3.We use the image processing functions from the Official version of ResNet. Please checkout the Models repository if you haven't
already, and add the Official Models to your Python path:
1
git clone https://github.com/tensorflow/models.git
2
export PYTHONPATH="$PYTHONPATH:/path/to/models"
Copied!
Step 2: Get a model to test
The provided script runs with the Official version of ResNet trained with ImageNet data, but can be used for other models as well, as long as you have a SavedModel or a Frozen Graph.
You can download the ResNetv2-ImageNet SavedModel or Frozen Graph, or, if you want to train the model yourself, pass --export_dir to the Official ResNet imagenet_main.py.
When running this script, you can pass in a SavedModel directory containing the Protobuf MetaGraphDef and variables directory to savedmodel_dir, or pass in a Protobuf frozen graph file directly to frozen_graph. If you downloaded the SavedModel linked above, note that you should untar it before passing in to the script.
Step 3: Get an image to test
The script can accept a JPEG image file to use for predictions. If none is provided, random data will be generated. We provide a sample image.jpg here which can be passed in with the --image_file flag.
Step 4: Run the model
You have TensorFlow, TensorRT, a graph def, and a picture. Now it's time to time.
For the full set of possible parameters, you can run python tensorrt.py --help. Assuming you used the files provided above, you would run:
1
python tensorrt.py --frozen_graph=resnetv2_imagenet_frozen_graph.pb \
2
  --image_file=image.jpg --native --fp32 --fp16 --int8 --output_dir=/my/output
Copied!
This will print the predictions for each of the precision modes that were run (native, which is the native precision of the model passed in, as well as the TensorRT version of the graph at precisions of fp32, fp16 and int8):
1
INFO:tensorflow:Starting timing.
2
INFO:tensorflow:Timing loop done!
3
Predictions:
4
Precision:  native [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty', u'lakeside, lakeshore', u'grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus']
5
Precision:  FP32 [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty', u'lakeside, lakeshore', u'sandbar, sand bar']
6
Precision:  FP16 [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty', u'lakeside, lakeshore', u'sandbar, sand bar']
7
Precision:  INT8 [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty', u'grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus', u'lakeside, lakeshore']
Copied!
The script will generate or append to a file in the output_dir, log.txt, which includes the timing information for each of the models:
1
==========================
2
network: native_resnetv2_imagenet_frozen_graph.pb,     batchsize 128, steps 3
3
  fps     median: 930.2,     mean: 934.9,     uncertainty: 5.9,     jitter: 3.3
4
  latency     median: 0.13760,     mean: 0.13692,     99th_p: 0.13793,     99th_uncertainty: 0.00271
5
​
6
==========================
7
network: tftrt_fp32_resnetv2_imagenet_frozen_graph.pb,     batchsize 128, steps 3
8
  fps     median: 1160.2,     mean: 1171.8,     uncertainty: 18.0,     jitter: 17.8
9
  latency     median: 0.11033,     mean: 0.10928,     99th_p: 0.11146,     99th_uncertainty: 0.00110
10
​
11
==========================
12
network: tftrt_fp16_resnetv2_imagenet_frozen_graph.pb,     batchsize 128, steps 3
13
  fps     median: 2007.2,     mean: 2007.4,     uncertainty: 0.4,     jitter: 0.7
14
  latency     median: 0.06377,     mean: 0.06376,     99th_p: 0.06378,     99th_uncertainty: 0.00001
15
​
16
==========================
17
network: tftrt_int8_resnetv2_imagenet_frozen_graph.pb,     batchsize 128, steps 3
18
  fps     median: 2970.2,     mean: 2971.4,     uncertainty: 109.8,     jitter: 279.4
19
  latency     median: 0.04309,     mean: 0.04320,     99th_p: 0.04595,     99th_uncertainty: 0.00286
Copied!
The script will also output the GraphDefs used for each of the modes run, for future use and inspection:
1
ls /my/output
2
log.txt
3
tftrt_fp16_resnetv2_imagenet_frozen_graph.pb
4
tftrt_fp32_resnetv2_imagenet_frozen_graph.pb
5
tftrt_int8_calib_resnetv2_imagenet_frozen_graph.pb
6
tftrt_int8_resnetv2_imagenet_frozen_graph.pb
Copied!
Troubleshooting and Notes
GPU/Precision Compatibility
Not all GPUs support the ops required for all precisions. For example, P100s cannot currently run INT8 precision.
Label Offsets
Some ResNet models represent 1000 categories, and some represent all 1001, with the 0th category being "background". The models provided are of the latter type. If you are using a different model and find that your predictions seem slightly off, try passing in the --ids_are_one_indexed arg, which adjusts the label alignment for models with only 1000 categories.
Model Links
​ResNet-v2-ImageNet Frozen Graph​
​ResNet-v2-ImageNet SavedModel​
​ResNet-v1-ImageNet Frozen Graph​
​ResNet-v1-ImageNet SavedModel​
README
Deepdrive
Last modified 2yr ago
Copy link
Contents
How to Run This Script
Step 1: Install Prerequisites
Step 2: Get a model to test
Step 3: Get an image to test
Step 4: Run the model
Troubleshooting and Notes
GPU/Precision Compatibility
Label Offsets
Model Links