Identifying and Counting Items in Real-Time with Fritz Object Detection for Android
Hi all! This post is a continuation of my earlier series: Machine Learning on Android using Firebase ML Kit.
In this blog post, I’ll be covering the steps to create an Android app that’s able to detect and count objects from the Camera Preview in real time.
There are tons of real-time object detection tutorials out there, so you might be wondering, How is this one any different? One key difference is that, instead of just detecting objects, we’ll also be counting objects in a given frame, opening up a much wider range of possible applications.
For example, imagine you’ve developed an app that suggests recipes to users based on the items they have at their disposal. By counting objects using camera-based, real-time object detection, you could empower your users to simply point the camera at the ingredients they have, and voilá!
Using an algorithm like this, the app would not only detect the ingredients but also identify the correct amounts needed for a recipe. So for instance, users could easily and seamlessly determine they don’t have enough apples for a pie or that they need more garlic for a delicious pasta sauce.
The magic of mobile machine learning has just made your users’ cooking or baking experience more convenient, more personalized, and more fun.
The best part about counting objects using real-time object detection is that inference happens on the device, without an active Internet connection (thanks to the Object Detection API provided by Fritz AI^). And the good news for us developers is, it won’t take us very long to do! We’ll try to finish the app in less than 5 minutes of coding!
Here’s a video showing what the finished app will look like:
Some Screenshots :
If that’s got you excited, then let’s start with the implementation part of the app!
Step 1: Create a new Android Studio Project and add the necessary dependencies
First, we need to create a new Android Studio project and add the relevant dependencies to it:
- Fritz Object Detection:
P.S. You’ll need to create an account at https://app.fritz.ai/login and register your app there first.
2. CameraView to get the camera preview:
Want to power your mobile app with object detection? Start building today with a Fritz AI account.
Step 2: Creating a basic layout and adding the camera preview
We need to create a basic layout in our app that hosts a Camera Preview and lets us interact with it.
Step 3 : Start the Camera Preview
This is super simple: go to your activity and start the preview in your onStart lifecycle method.
At this point, if you run the app, you should see a screen that shows you a continuous preview of your Camera.
Next up, we’ll be adding a method that gives us these preview frames so we can perform some machine learning magic on them!
Want to dive deeper into mobile machine learning? Subscribe to the Fritz AI Newsletter to receive our expert takes on all things mobile ML, delivered right to your inbox.
Step 4 : Add a FrameProcessor to your CameraView to get the live frames from the preview
On our CameraView, we can add a FrameProcessor that gives us the captured frames and lets us perform ML inference on those frames.
The code for doing that is pretty straightforward:
Step 5 : Convert the frame to a Bitmap
This step is somewhat tricky because the frame that we receive contains the ByteArray for preview, but that ByteArray is of NV21 format.
The Fritz SDK reads only bitmaps, so we need to convert this ByteArray to a Bitmap readable format.
The following code does this, but in case you want to know more about the process, please check the following Stack Overflow discussion thread.
Step 6 : Perform inferencing on the converted Bitmap
To get this up and running, we first need to create an instance of the FritzVisionObjectPredictor class we’re using. You can do that easily, as follows:
Next up, we need to create a FritzVisionImage from the Bitmap that we just converted above. The code for this is super simple as well:
Lastly, we need to run this FritzVisionImage through the FritzVisionObjectPredictor we created earlier:
And that’s it!
If you run the app now, you should see that your app is able to detect and identify the objects in the preview frame in close to real time. Isn’t that cool?!
The Object Detection API provided here can detect over 90+ daily use items from an image and also runs without Internet connectivity, so it’s a perfect tool for someone looking to add a bit of smartness to their apps 😉
The complete Kotlin code for the Activity can be found below:
The app shown in the video above can be found in my GitHub Repository:
Thanks for reading! If you enjoyed this story, please click the 👏 button and share to find others! Feel free to leave a comment 💬 below.
Have feedback? Let’s connect on Twitter.
Do you want to start building similar amazing Android Apps? Check out my course on Coding Bocks .
^Disclosure: Heartbeat is sponsored by Fritz AI
Discuss this post on Hacker News
Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.
Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.
If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Fritz AI Newsletter), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.
