How to use Tensorflow Graph with OpenCV DNN module

In this tutorial we will see how we can use a pre-trained Tensorflow module in OpenCV DNN module. OpenCV ≥ 3.4.3 has a module Deep Neural Netowork , which can be used for inference using a pre-trained model. Framework supported by OpenCV are Caffe, Tensorflow, Darknet and PyTorch. But you can not directly use the Tensorflow pre-trained model with OpenCV DNN module. Below we will see what are the steps we have to follow to make the Tensorflow module work with OpenCV DNN module.

System Specifications:

1. Tensorflow 1.13
2. OpenCV 4.0.1

Tensorflow Model Description:

For this article, I have used my pre-trained OCR model. Layers used are

1. Dataset and Iterators ( I have used OneShotIterator)
2. Placeholders
3. Convolutional Layers with Batch Normalization
4. MaxPool
5. Fully Connected Layers with Batch Normalization
6. Dropout
7. Activation function Relu, Softmax

OCR Model (Taken from Tensorboard)

Steps needed are:

1. Freezing graph
2. Transforming graph
3. Replacing placeholders with constants
4. Optimizing batch normalization blocks
5. Removing dropout blocks

Step 1: Freezing the Graph

After you finished training your model, you will have .meta and checkpoint files. As you can see from the above graph, we have lots of trainable variables and nodes in our graph. These all are not needed during inference. So we can remove them from our graph. Also we will convert the checkpoint variables to Const operation. We will use freeze_graph.py tool provided by Tensorflow.

python3 freeze_graph.py --input_meta_graph alpha_digit_ocr_engine-171439.meta --input_checkpoint alpha_digit_ocr_engine-171439 --output_graph frozen_inference_graph.pb --output_node_names predicated_output --input_binary True

Frozen Graph

Step 2: Transforming Graph

After freezing the graph, we still have some nodes which is specific to Tensorflow like dataset, iterators, training block of batch normalization and are not needed during inference. We will use Graph Transformation tool provided by Tensorflow.

Script for graph transformation

In TransformGraph, give the input and output node correctly so that it will only keep the nodes related to the input and output.

We have applied the following transformation.

1. strip_unused_nodes : Remove unused nodes for inference
2. remove_nodes: Remove any specific nodes like Identity nodes.
3. fold_constants: Looks for any sub-graphs within the model that always evaluate to constant expressions, and replaces them with those constants.
4. fold_batch_norms: Optimize the batch normalization block. This transformation is needed if you have used batch normalization during training.
5. fold_old_batch_norms: For old batch normalization.
6. sort_by_execution_order: Arrange the nodes in topological order. This is an important transformation because OpenCV will parse the graph in topological order. And if the graph is not in topological order, it will throw some missing nodes error.

Batch Normalization block before Transformation

Batch Normalization block after transformation

Step 3: Replacing placeholders with constants

We should have only one input placeholder. So we will replace other placeholders like keep_prob (used in dropout), is_training (used in batch normalization) with Constant node.

Step 4: Optimizing batch normalization block

Not all the Tensorflow ops (operations) are defined in OpenCV. Ops like Switch and Merge (used inside batch_normalization), RandomUniform (used inside dropout) are not defined in OpenCV. So we have to inspect all those operators and simplify those blocks.

Batch Normalization block

Once the training is done, we don’t need such a complicated block for inference. We can replace the whole batch normalization block with a FusedBatchNorm ops. We also have to change the input to this node. And the input to the next Convolutional node or MaxPool node should be the output of this node.

Batch Normalization block for fully connected layer

Batch norm block in Fully connected layer is little different from the one present in Convolutional layer. We can remove moments, cond_1 and cond block and change the input of batchnorm block.

Important: Before proceeding to next step, call transform_graph() function defined in step 2. This will further optimized the batch normalization node.

Step 5: Removing dropout block

Dropout is only needed during training. During inference, we pass 1 to keep_prob which make sure all the neurons are present. So we can remove the dropout layer and directly connect the output of fully connected layer fc1 to input of fully connected layer fc2. Dropout block has an operation RandomUniform which is not defined in OpenCV. So this will throw error.

Finally call the transform_graph() to make sure all the constants are folded and more importantly to ensure the graph is in topological order. The graph file generated by the last transform_graph() is used in OpenCV DNN module.

Testing:

Complete Code:

Important Points:

1. Make sure the graph is in topological order.
2. Freeze the graph to convert variables to constants.
3. Tensorflow operations defined in OpenCV DNN module are Conv2D, SpaceToBatchND, DepthwiseConv2dNative, Pad, BiasAdd, Add, Sub, MatMul, Reshape, Flatten, Squeeze, Transpose, Const, LRN, Concat, ConcatV2, MaxPool, AvgPool, Placeholder, Split, Slice, Mul, FusedBatchNorm, Conv2DBackpropInput, BlockLSTM, ResizeNearestNeighbor, ResizeBilinear, L2Normalize, PriorBox, Softmax, CropAndResize, Mean, ClipByValue, Abs, Tanh, Sigmoid, Relu, Elu, Identity, Relu6.
4. Make sure your graph contains operations mentioned in point #3.
5. If you have some operation which are not present in #3, check if those operations are training specific. If yes, remove them from graph.
6. Use tensorboard to visualize your graph and try to optimize it where possible.

Conclusion:

In this article, we learned how can we use a pre-trained Tensorflow graph with OpenCV DNN module. Also it will optimize the graph which will reduce the inference time. I hope you enjoyed this long article and learned something new and useful.