On-Device AI — What I know so far
We are in the dawn of the acceleration of Artificial Intelligence applications. Processing tasks in AI algorithms in these applications take place on either cloud-based infrastructure or on device. Besides the cloud-based approach, on-device approach is getting trending nowadays due to increased privacy, low latency, increased reliability and many other advantages.
By 2022, 80 percent of smartphones shipped will have on-device AI capabilities, up from 10 percent in 2017 - Gartner
What is on-device inference?
On-device inference is the process of making predictions using a trained model which will run on the device. On-device inference is becoming more popular due to its lower latency and increased privacy when compared to cloud-based paradigm. However, executing such compute-intensive tasks on small devices can be harder due to lack of computation power and energy consumption.
What is on-device training?
Answer is in the question! 😂 Training your model on the device. Again, just like on-device inference, major challenges of using on-device training are computational power limitation on such devices and energy consumption. But on-device training gives more advantages. Your model can learn from user’s data. Since the model runs on the device, it can easily get user’s behavior. Besides that, it can be personalized to that user. Since training happens in the device, there is no need to upload these data to cloud. So, it guarantees data privacy. Also, there is no need to host a server to train your model, so it saves your money also.
In this article, I point out some useful and popular resources for newbies related to On-Device AI I found on the internet. I will keep this article updated. Please comment if you know missing information, so I can update this list. 😁
Mobile apps which do training/inference on mobile
Here are some popular mobile apps/features which do on-device inference and training on your mobile.
01) The “Hey Siri” feature — You can find how “Hey Siri” works in this article.
02) “Now Playing” feature on Pixel phone which identifies the music being played around you — You can find the research paper related to that research here
03) Face ID technology on iPhone — To find how it works refer this article
04) The Photos app on Apple devices process your images, recognize face and places does on the device — Reference
Tensorflow Lite — Tensorflow Lite is an open source deep learning framework which supports on-device inference. On-device training is currently not supported by TensorFlow Lite. To use Tensorflow Lite on your application, first you have to convert your Tensorflow model into a compressed flat buffer using TensorFlow Lite Converter. It will create a .tflite file. After that, you can load it into a mobile or embedded device and run the model on the device using TensorFlow Lite interpreter.
PyTorch Mobile — Currently, PyTorch Mobile is in beta. Unlike Tensorflow Lite, it doesn’t need to convert the existing machine learning model into an intermediate file format.
ML Kit by Google — ML Kit is a mobile SDK which is currently in Beta release.
You can join to ML Kit’s early access program using this link.
Core ML by Apple — One of the cool features of Core ML is you can convert models from other machine learning libraries like TensorFlow and PyTorch to Core ML using Core ML converters.
Embedded Learning Library (ELL) by Microsoft — EEL is mainly designed for resource constrained platforms and small single-board computers such as Raspberry Pi.
ML Kit by Huawei — ML Kit provides both on-device and on-cloud APIs.
Samsung Neural SDK— Samsung Neural SDK provides APIs allow developers to easily deploy on-device pre-trained or custom neural networks. It is designed to run only on Samsung devices.
Processors which support On-Device AI
So far, what I discussed about in this article is brief intro to on-device inference and training and some popular examples. I will be here again once I learn some new things. 🤗