Real-Time Machine Learning With Flutter Camera
My recent assignment was to do real-time ML with Flutter, in which I was met with a lot of issues. After spending around one month working on it, I decided that I should write a blog for myself in the future, and for anybody that has to do something similar. I hope this article could save someone time so that they don’t have to research and look for ways to implement it. I will tell you the whole story: from the first ML attempt to the final version that hopefully would work for all cases.
TL;DR Scroll down to the bottom of the article to grab a sample code and the summary of what I’ve done
My task was to implement face liveness detection in Flutter. This process is meant to check the liveness of a selfie. In the old days, we were required to carry a citizen ID and meet with a bank teller at the physical branch in order to open a bank account. Now, barely anyone go to a bank to do this. The in-person meeting is replaced with the eKYC process. The digital system compares the face to the one on your ID to confirm if it’s the same person. Meanwhile, we need to make sure that the facial image sent to us is a real face, not a still photo. That’s why we need ML on the device to detect this. We have to get the image, send it to ML, validate the image, and respond back to Flutter.
The First Version
I took a photo by setting a timer to take a selfie every one second, then converted those to Uint8List, binary in Flutter, and sent it to Native to rebuild an image for feeding into ML. This first version was only working well for high-end devices because as you take a photo, the system needs about 300–400 ms to send the photo to Flutter, with about 600 ms left before the next photo is taken. My iPhone 11 Pro Max took about 300–400 ms, while iPhone X took about 1,200 ms, which was not fast enough to be considered as real-time. Moreover, iOS has a shutter sound as you take a selfie. This creates a bad experience for users when they do facial verification. Nevertheless, this version is okay to pass along to the tester to test the face liveness feature.
The Second Version
Since taking a series of photos was not fast enough, I decided to get the stream image and feed it to ML instead.
In Flutter camera https://pub.dev/packages/camera, there’s a function called startImageStream that streams an image to Flutter.
controller.startImageStream((cameraImage) async {
// Feed image into ML});
This will return CameraImage. When it comes to ML images, most people would probably use Firebase ML for Flutter, which already accepts CameraImage. You can feed that into ML and get the result back to display in the UI.
In some cases, your requirement might not match Firebase ML and you have to implement your own model. Of course, most ML models only support Native, either Swift or Kotlin, meaning you must send the camera image from Flutter via the Flutter method channel to Native, do some tasks over there, and send the result back. Here’s the problem with that.
Most SDK requires you to feed RGB format, either JPG or PNG. In most cases, we use JPG because it’s smaller. CameraImage also supports JPG, but only in Android, not iOS. I began by setting up Android to receive a JPG image. For iOS, I read camera documents and found out that they support two formats: YUV420, and BGRA8888. I decided to go with BGRA8888 because it’s more like an image while the other one is more like a video.
Android is already a JPG, so I could just feed it to the same ML without any problems. It’s iOS that I had to figure out how to convert those two formats into JPG. I did some research and found this.
I followed this guideline and got a JPG back, but I was not happy with the result as it was still pretty laggy. I need to feed it constantly, so I have to convert it all the time. The above gist should be fine with just a single image but not with the real-time face liveness.
To rectify this, I added a throttling function in Flutter. For every camera image sent from Flutter, I would convert it to an image only once every 500 ms. Flutter feeds the image back every 20–30 ms. After trials and errors, I felt that 30 ms and 500 ms were not that different for my use case. Since our face stayed in the middle of the screen and doesn’t move that fast, the user wouldn’t feel any delay.
After I hit the wall with iOS, I decided to shift to Android instead, figuring I should make it work for at least one platform so that later on, I could pour all my focus on the other. In Android, the JPG flag worked well, and the performance was quite good. Though, I seemed to have a little problem with the images that I sent to Kotlin: they ended up being rotated 90 degrees. To be honest, I don’t understand this at all. Why would you have to send a 90-degree image back to us from Flutter? 😢 Anyways, they must have a reason for this since it’s a lib that people around the world are using. I ended up using Kotlin code to create Bitmap and rotating the image 270 degrees with Matrix before feeding it into ML. Medium-tier Android still lagged a bit since the entire process of building images, rotating images, and converting to binary used up a lot of power. That said, it was still better than taking a series of photos and send to ML. High-tier Android was working fine, medium-tier lagged a bit, and low-tier was unusable. Still, it was good enough at the time.
Going back to iOS, I came up with the idea of threading in Flutter. I followed the instruction in this article.
Usually, I barely need any threading in Flutter because await function is not blocking the thread, while for iOS, we have to create threading and constantly put extensive tasks into the thread.
I decided to use threading in Dart for the first time. After following the article above, I got a sample with decodeImage and used Image lib from Flutter to convert BGRA to RGB.
Wow! The result was completely different. My iPhone X was, as BTS would say, smooth like butter 😄
Now that my second version proved that it could support mid-tier to top-tier iOS and Android, it was time to test with a wider audience.
After wider testing, I have found a critical problem: the JPG flag in ImageFormatGroup.jpeg is not supported on some devices!!!
I encountered this error GetYUVPlaneInfo: Invalid format passed: 0x21
Out of 20 Android devices, only Xiaomi Note 8 crashed when trying to stream an image for JPG. I don’t know if this issue only occurred for this model or if there would be other devices that had this problem had I launched this feature into production.
After some investigations, it seems that if I change to ImageFormatGroup.yuv420, it wouldn’t create any problem. I didn’t want to risk and push the feature into production, so I decided to go with YUV420 instead, which is the default format for Flutter. Doing some research, I got this code off the internet. With this, I could send the CameraImage via method channel to android to create an image.
List<int> strides = Int32List(image.planes.length * 2);int index = 0;List<Uint8List> data = image.planes.map((plane) { strides[index] = (plane.bytesPerRow); index++; strides[index] = (plane.bytesPerPixel)!; index++; return plane.bytes;}).toList();await _channel.invokeMethod<Uint8List>("checkLiveness", { 'platforms': data, 'height': image.height, 'width': image.width, 'strides': strides});
Disclaimer: I really want to credit the original creator of this code above, but I’m unable to find the source. If anyone knows where to find it, I’m happy to add their credit to this article.
As for Android, I got the code for converting YUV to JPG from this page.
After being implemented, the feature was working fine, even though it was still lagging since what we’d done so far was to solve the JPG problem in Android.
Now it’s time to improve the performance for a better user experience.
I researched the Kotlin coroutine to do the threading job and applied it to my Kotlin code to make the process smoother. The result was positive with Android getting a lot smoother and working fine on all tiers. Even the lowest tier is still smooth. My Oppo a3s can do liveness with a very minor lag.
The Final Version
This time I investigated low-tier iOS: iPhone 6s, 6s Plus, and 7.
I found out that even Isolate was working fine with no lagging for medium-tier, but not for low-tier. For Isolate to finish their task, it took about 1- 1.5 seconds, and another 1.5 seconds for liveness detection, which was too much for users to handle on slower devices. I decided to move to native instead, and the performance there exceeded my expectation. It went from 1.5 seconds down to about 0.01 seconds!!! I shared the code to convert it into the repository at the end of an article. Here’s the sample code to convert images.
private func bytesToPixelBuffer(width: Int, height: Int, baseAddress: UnsafeMutableRawPointer, bytesPerRow: Int) -> CVBuffer? { var dstPixelBuffer: CVBuffer? CVPixelBufferCreateWithBytes(kCFAllocatorDefault, width, height, kCVPixelFormatType_32BGRA, baseAddress, bytesPerRow, nil, nil, nil, &dstPixelBuffer) return dstPixelBuffer ?? nil}private func createImage(from pixelBuffer: CVPixelBuffer) -> CGImage? { var cgImage: CGImage? VTCreateCGImageFromCVPixelBuffer(pixelBuffer, options: nil, imageOut: &cgImage) return cgImage}private func createUIImageFromRawData(data: Data, imageWidth: Int, imageHeight: Int, bytes: Int) -> UIImage? { data.withUnsafeBytes { rawBufferPointer in let rawPtr = rawBufferPointer.baseAddress! let address = UnsafeMutableRawPointer(mutating:rawPtr) guard let pxBuffer = bytesToPixelBuffer(width: imageWidth, height: imageHeight, baseAddress: address, bytesPerRow: bytes), let cgiImage = createImage(from: pxBuffer) else { return nil } return UIImage(cgImage: cgiImage)}
The final problem and the last big wall: Flutter image stream.
I tried creating a blank Flutter project, implemented the Flutter camera, and opened the image stream without doing any tasks. Even so, it was still lagging for low-tier iPhones. If I turned off the image stream, the preview became smooth like other iPhones. But how I can do liveness detection without getting an image from the image stream?
Eventually, I came up with a way to hack this problem: by starting the image stream and turning it off after 50 ms. If observed closely, Flutter app seems to lag for 50 ms, but in most cases, the user won’t be able to feel it. In those 50ms, I got 1–2 images back from the Camera image, which I then fed into ML.
After implementing it, despite a tiny lag, my iPhone 7 Plus could do face liveness detection. As for iPhone 6s and 6s Plus, they became a lot better. You can see the result below.
Performance Inspection
iPhone 11 Pro Max — 53% in CPU, 190MB for Memory, the image returns every 20–40 ms
iPhone 6s — 118% in CPU, 160MB for Memory, the image returns every 20 — 30 ms
iPhone 6s Plus — 138% CPU, 187 MB for Memory, the image returns every 20–50 ms
iPhone 6s Plus without the camera stream, open only camera preview, consuming 46% CPU
iPhone 6s Plus with the stream turned on and off, it’s going up and down between 50% to 70%
For Android, my low-tier device, Oppo A3s (2018), Andriod 8.1.0, Ram 2GB, is doing better than my iPhone 6s. I open the image stream at all times and it consumes only 14% of CPU, and around 320 MB of Ram.
Meanwhile, iPhone 6s Plus with a Native camera consumes only 57% CPU, 39.1 MB.
Everything finally works fine for medium-tier iOS and all-tier Android. I have one issue left with UIImage from the code that I share above. When liveness returned the image we needed to use, I got an error about Thread 1: EXC_BAD_ACCESS
I googled CGDataProvider_BufferIsNotBigEnough to see how to fix it, but couldn’t find anything related to my issue. Some mentioned the image was too big or your phone is outdated! Even my fastest device, iPhone 11 Pro Max, still had an issue, so that was definitely not it. I tried many ways such as feeding smaller images and feeding images to liveness much slower, but none succeeded. I was totally blank. I had no idea what it is. Is it the liveness or my code? After a couple of days of investigating with trails and errors, I finally realized it was the images I fed into SDK that returned when liveness was successful. That image was sent in reference type instead of value type. I then looked up how to copy the reference type into the value type, and found the deep copy function for PixelBuffer. I added it to my project, and it finally solved the problem of memory access. Everything is fine now, phewww
Final Verdict!!
TL;DR Don’t do Real-time ML in Flutter, have all the processes done in Native view and send the result back to Flutter. It’s a lot easier than trying to hack around to make it work in Flutter.
Flutter is supposed to reduce our development time. If it can’t serve our purpose, we just do it natively. No big deal, it’s not a silver bullet that solves every problem 😁 I believe that the time I spend on Flutter to solve all those problems can be solved using native instead.
Updated: 15 Dec 2024
I have revised and recreated the code to work natively instead of using the Flutter approach. You can check out the details on Medium below.
For anyone still interested in using the less complex method, you can continue to follow this article
Here’s the mocking sample project: I’ve implemented everything I learned from my final versions. If anyone wants to keep doing it in Flutter, feel free to check this out.
This is the video where I use Flutter stream to show the performance comparison.
After implementing everything with our internal face liveness detection in the app, here’s the final result.
Summary
Flutter camera image is unsuitable for low-end devices because it constantly sends stream images back to Flutter. I believe that if Flutter camera can have a parameter for frame rate that they can return, It would help a lot. Even if I did throttling, Flutter would still have to process overhead images that I didn’t need. Instead of getting an image every 20–30 ms, If I could get an image every 200 ms instead, that would reduce the work by 90%!!!
This request has been open for over 2 years now.
I don’t think it will happen any time soon, so let’s do native for now. I would like to open MR to add functionality to this camera but let’s see If I have a chance to do it.
Anyone interested in implementing face liveness in their project, either in Flutter or Native, you can contact KBTG or me. I will forward your request to our team. Here’s our KBTG Face Liveness, ISO 30107–3 tested by iBeta.
Reference website:
Want to read more stories like this? Or catch up with the latest trends in the technology world? Be sure to check out our website for more at www.kbtg.tech
