« Mobile First to AI First » : Machine Learning inside our apps

[This article is originally published in French on the Octo Technology Blog]

In October 2017, Droidcon UK hosted a very interesting talk: My app is smarter than your app d’Erik Hellman. The speaker talked about the importance of Machine Learning and how much it’s time to accept it in the mobile world. Personnalisation, recognition, user assistance… There are many usages. Various API give access to this intelligence remotely. But in this world where smartphones are becoming powerful, why not work on devices directly? This is why, I need to improve TensorFlow.

TensorFlow ?

Machine Learning is a buzzword nowadays. It’s about giving a machine the ability to learn, to derive patterns from a dataset. This same machine will be able to analyze other dataset to draw conclusions.

TensorFlow is an opensource library for dataflow programming. Initially, Google is developing it for research on Machine Learning.

The Tensorflow website gives some usages: image, optical character, voice and gesture recognition, object localization and text classification. Currently, this library is used in AirBnb, Ebay and Twitter.

How it works

TensorFlow website and documentation aren’t the most explicit. So, three codelabs TensorFlow for Poets are available for a first introduction. This codelabs deal with the construction of the model to its integration in an Android app. They are all based on the image recognition capability.

The codeLab contains a python script that uses TensorFlow. From a flower dataset sorted by species, the script infers patterns and generate two files: it is the learning phase. The first file contains labels of each species. The second one is the model. This model can deduce a flower species from a flower picture (if this flower specie is in the labels file).

Then, we are invited to push the flower model in an Android app. This app films flowers and can deduce the flower species (prediction). In fact, the python script uses TensorFlow to learn and produce a model whereas the app uses TensorFlow to read the model and find an answer.

Looking at the code, we realize that the content of the model (here, the prediction of flower species) is independent of the application purpose. So, if the model change (with other things than flowers), the app still works. To test this hypothesis, I took about a hundred pictures of corgis. There are two corgi species: Pembroke and Cardigan. I’ve classified these pictures according to these two species.

I ran the python script (the one given by the CodeLab, with TensorFlow inside) that provided a new model and a new labels file. I’ve pushed this new model in an Android app and reloaded in my device.

The result is quite bluffing: TensorFlow predicts a solution with 99% of accuracy. And, I confirm, it’s really a beautiful Pembroke.

But as a Mobile Developer, it’s the TensorFlow use cases through Android that interests me here.

Include TensorFlow in ours Android Apps

Today, the most simple way to include Machine Learning in an application is to make API calls. Google Cloud Platform abounds of images, videos and texts recognition. We can also host our own TensorFlow solution in a server.

But the network quality isn’t stable. There are many situations where the network is insufficient or missing. A server — mobile exchange can be slow, or not quick enough for you. It’s for this kind of issues that integrating TensorFlow directly into devices can be the solution.

Google has two Machine Learning solutions on mobile platform; TensorFlow Mobile and this new little brother, TensorFlow Lite.

TensorFlow 1.0 was announced the 15 February 2017. It contains its mobile version: TensorFlow Mobile. This version is complete: we have access to all TensorFlow abilities.

TensorFlow Lite arrived the 14 november 2017. It’s considered like a TensorFlow Mobile evolution. Lighter and faster than its predecessor, it is just a preview and own only the main TensorFlow functionnalities.

Google recommend to use TensorFlow Mobile in production. In the TensorFlow Github project, the Lite version is already considered like a contribution.

To understand these two versions, the TensorFlow for Poet 2 codelab contains two versions of the same application. One use TensorFlow Mobile, the other TensorFlow Lite. In both cases, the model is previously trained in desktop (learning), before being pushed in the app (where it will only be predictive).

Here, I dismembered the code toextract only the basic principles and compare their ease of use.

Integration

TensorFlow Mobile

We just need to add the TensorFlow Android library from the jcenter:

repositories {
    jcenter()
}
dependencies {
    compile ‘org.tensorflow:tensorflow-android:1.4.0’
}

TensorFlow Lite

TensorFlow Lite is in preview yet. We need to take it from Bintray:

android {
    […]
    aaptOptions {
        noCompress “tflite”
        noCompress “lite”
    }
}
repositories {
    maven {
        url ‘https://google.bintray.com/tensorflow'
    }
}
dependencies {
    compile ‘org.tensorflow:tensorflow-lite:0.1.1’
}

The property noCompress inside the aaptOptions block specifies that tflite and lite extensions should not be compressed when the APK is create.

Conclusion

Integration is simple from one side to the other, even if TensorFlow Lite is still in preview.

Implementation

TensorFlow Mobile

TensorFlow Mobile need a Protocol Buffer model. This model is trained on desktop and then integrate in the Android project assets.

Here, there is the TensorFlowInferenceInterface object. It’s initialize with the model name, present in the assets (called graph.pb in this example):

val modelFileName = “file:///android_asset/graph.pb”
inferenceInterface = TensorFlowInferenceInterface(assetManager, modelFileName)

Before running the prediction, we need to feed our TensorFlowInferenceInterface object with the image we want to analyze. A bitmap picture is transformed in pixels array, and transferred in a float array (imgValues):

bitmap.getPixels(pixels, 0, bitmap.getWidth(), 0, 0,    bitmap.getWidth(), bitmap.getHeight());
for (int i = 0; i < pixels.length; ++i) {
    final int val = pixels[i];
    imgValues[i* 3 + 0] = (((val >> 16) & 0xFF) — imageMean) / imageStd;
    imgValues[i * 3 + 1] = (((val >> 8) & 0xFF) — imageMean) / imageStd;
    imgValues[i * 3 + 2] = ((val & 0xFF) — imageMean) / imageStd;
}

The float array (imgValues) feeds the TensorFlowInferenceInterface object:

inferenceInterface.feed(inputName, imgValues, 1, size, size, 3);

The run command run the image prediction contained in imgValues thanks to graph.pb model:

inferenceInterface.run(outputNames);

Finally, thanks to fetch command, the probabilities of each label contained in the model are injected in the float array outputs.

inferenceInterface.fetch(outputName, outputs);

We can compare the results with the labels array (label.txt), and extract, for example, the most likely solution:

labelList
    .mapIndexed { id, label -> Pair(label, outputs[id]) }
    .maxBy { it.second }?.first ?: “Unknown”

TensorFlow Lite

Unlike TensorFlow Mobile, TensorFlow Lite want a model in Flat Buffer format. A converter is proposed to transform a Protocol Buffer model in a Flat Buffer Model.

The TensorFlow Lite classifier is called Interpreter. It’s constructed with a ByteBuffer containing the model (read from graph.lite):

val modelFile = activity.assets.openFd(“graph.lite”).let {
    FileInputStream(it.fileDescriptor).channel.map(
        FileChannel.MapMode.READ_ONLY,
        it.startOffset,
        it.declaredLength
    )
}
tflite = Interpreter(modelFile);
tflite = Interpreter(modelFile);

Then, TensorFlow Lite establish a result with just one command. The bitmap is transformed in a pixels array and injects in a ByteBuffer. The interpreter takes this ByteBuffer as first parameter, and an empty float array (outputs) as second parameter. The command “run” fills the outputs array with each label probability (contains in label.txt).

val newBitmap = Bitmap.createScaledBitmap(bitmap, SIZE_X, SIZE_Y, false)
newBitmap.getPixels(pixels, 0, newBitmap.width, 0, 0, newBitmap.width, newBitmap.height)
pixels.forEach { pixel ->
    imgData.putFloat(((pixel shr 16 and 0xFF) — mean) / std)
    imgData.putFloat(((pixel shr 8 and 0xFF) — mean) / std)
    imgData.putFloat(((pixel and 0xFF) — mean) / std)
}
tflite.run(imgData, outputs)

At the end, the labels arrays is crossed with the outputs probability array. Here, we can have the most probably label:

labelList
    .mapIndexed { id, label -> Pair(label, outputs[0][id]) }
    .maxBy { it.second }?.first ?: “Unknown”

Conclusion

This two Android TensorFlow versions are pretty simple to use. But we can give the victory to TensorFlow Lite thanks to its ease to use: TensorFlow Mobile needs three steps to have a result (Feed, Run, Fetch). TensorFlow Lite needs just one (Run).

Performance

The two application versions were tested in cold start using a Sony Xperia with Android 7.1.1.

The results are in favor of TensorFlow Lite with a lightweight APK and a faster and costless prediction.

Conclusion

TensorFlow Lite has, as promised by Google, better performance and is easiest to use. However, this version is only in preview, and is considered more like a contribution than a real library. So, it’s difficult to imagine TensorFlow Lite in production. But we need this version because TensorFlow Mobile is very greedy for a smartphone battery.

TensorFlow, and generally Machine Learning, offers lot of possibilities. We can imagine an application with an extreme personalization. But Google present essentially examples with image recognition.

In an offline environnement or quick interaction with user, integrate TensorFlow directly in devices can bring a new dimension to our apps. The use cases are still few but we are convinced that issues related to the intelligence of our smartphones will develop soon.

This article was a TensorFlow introduction as an Android Developer. In a second part, I’ll present to you a TensorFlow Mobile (or Lite?) app with a real use case.