TensorFlow Lite 2.x. SWIFT or C API for iOS deployment?
TensorFlow Lite 2.x API allows us to deploy neural network models on iOS devices. TensorFlow Lite comes with three different programming languages support: SWIFT, Objective-C, and C. However only the two first are recommended on the official TensorFlow website. In the following article, I describe differences and reasons why I prefer using TensorFlow lite C API, instead of recommending SWIFT or objective-C languages. Moreover, I provide code examples for SWIFT and C API models implementation.
TensorFlow Lite 2.x deployment in the Xcode environment
Deployment instruction includes the TensorFlow Lite and OpenCV framework installation. OpenCV will be used for image pre-processing while using C API.
Pre-step
It is required to have CocoaPods installed on the Mac. It can be installed by following the instruction on the official website.
Installation
- Create a new project in Xcode.
- Close the Xcode window.
- Open a terminal window in the project directory.
- Run ‘pod init‘. A pod file will be generated in the project directory.
- In the generated file add the following line for Swift version:
pod 'TensorFlowLiteSwift'- or following lines for C version of TensorFlow lite:
pod 'TensorFlowLiteC'
pod 'OpenCV2'
A comparison between the usage and the performance of both languages is presented below in the article.
- Save the file and run the command ‘pod install’. If you made a mistake and correction to the pod file is necessary run ‘pod upgrade’ instead of ‘pod install’. After successfully running the command, a *.xcworkspace file is created. Open the project by double click on the *.xcworkspace file.
Feeding image into the TensorFlow Lite model — SWIFT.
Usually, images on iOS come in two different types: UIImage or CVPixelBuffer. The input to the TensorFlow Lite model is a float array, where each array value represents a next pixel value. Moreover, Apple uses 4 channels for image coding: BGRA or RGBA, where the first 3 channels correspond to colors and the last Alpha channel is responsible for opacity. In order to insert the image into the TensorFlow Lite model, it is necessary to convert the Apple image type into a float array.
UIImage to CVPixelBuffer
Usually, the CVPixelBuffer is already available and there is no need for UIImage to CVPixelBuffer conversion. CVPixelBuffer is a standard output type from the iPhone camera. However if only the UIImage is available, conversion to CVPixelBuffer is necessary.
Create the UIImage class extension and then use code below to convert UIImage to CVPixelBuffer.
UIImage(ciImage: CIImage(cvPixelBuffer: imageBuffer))CVPixelBuffer to TensorFlow Lite model input
The official TensorFlow recommendation is to convert CVPixelBuffer to the input ‘tensor’ using their function presented below. The function copies the CVPixelBuffer to the float array, skipping the alpha channel in the CVPixelBuffer.
After creating a Data buffer (which is a float array), the input can be fed into the model and inference can be performed.
Feeding image into the TensorFlow Lite model — C.
Previous paragraph presented necessary steps for running a TensorFlow Lite model on an image using SWIFT. The same result (but faster) can be obtained using TensorFlow Lite C API. For the purpose of running inference using C language, we will use OpenCV for image pre-processing. The OpenCV framework was already installed in the installation step.
In order to perform inference on an image using TensorFlow Lite C API, it is necessary to create a float array from an image. The easiest way to do it is by creating cv::Mat from the UIImage or CVPixelBuffer. As the OpenCV library is available only in C, we will use Objective-C and C++ code.
UIImage to cv::Mat
The UIImage can be converted to cv::Mat using standard OpenCV function UIImageToMat().
CVPixelBuffer to cv::Mat
cv::Mat can be created from CVPixelBuffer using the function presented below.
cv::Mat stores the image already in one place in the memory. The only necessary action is creating a float pointer pointing to the first element of the Mat (first pixel) and using TensorFlow function to copy the whole frame into the model input. Voila! The inference can be performed.
Why do I prefer TensorFlow Lite C API, rather than TensorFlow SWIFT API?
So far we saw 2 different approaches to run the inference on images using TensorFlow Lite API. One using C API and second using SWIFT API. Now let's evaluate both programming languages in terms of processing time.
As it is shown in the table above, creating an input tensor for the TensorFlow model is x62 faster in the case of TensorFlow Lite C API. Further model inference time is similar, but the pre-processing time is way slower in the case of SWIFT.
Apart from that, having a cv::Mat allows us to perform advanced image pre-processing using OpenCV library. OpenCV library in many cases performs better, then SWIFT library for CVPixelBuffer handling.
TensorFlow Lite vs CoreML framework
One can ask whether to use TensorFlow Lite if the CoreML framework is available. The CoreML framework was specially designed for Apple architecture. There might be a couple of reasons:
- Avoiding model conversion from the TensorFlow model to the CoreML model. In some rare cases, the conversion might not be possible due to a lack of special types of layers or connections.
- Using CoreML requires to deal with MLMultiArrays. It is a CoreML version of a tensor. However, operations on MLMultiArrays are slow, especially if the array is large. Thus for image processing, Apple decided to use CVPixelBuffer as the input to the network. To accept CVPixelBuffer as the input, the model needs to be specially converted. CVPixelBuffer as the model input doesn’t support however custom inputs. For example images with 2 channels or 5 channels. CoreML supports only standard image shapes of shape BGRA. Thus TensorFlow Lite might perform better in particular situations.
- code generalization — code written using TensorFlowC can be easily moved to the Android implementation (and vice versa).
In general, the CoreML should perform better and should be used if possible. On the other hand, TensorFlow has Metal API support (Apple GPU support) and inference speed using TensorFlow can be comparable to CoreML. Each case is unique and should be judged on its own merits.
