Creating a Pet Sticker App on Android with Fritz Pet Segmentation

Image Source

The more a phone’s camera understands, the more powerful and creative mobile applications can be. Few subjects are more important to understand than our pets: doggos, floofs, fluffers, clouds, woofers, and even cats (or so I’m told).

In this post, we’ll learn how to use Fritz, a mobile machine learning platform for iOS and Android, to give an Android app the ability to generate pixel-perfect segmentation masks of pets in photos and video.

We’ll use this mask to create fun stickers you can share in apps like WhatsApp or Messenger, but you could also use it to track their growth over time or change the color of their fur.

Defining the task

There are many tasks we could teach our phone’s camera to perform: identify a breed, locate an animal’s exact position, check if a dog is sitting down, etc. In our case, we want to know exactly which pixels in an image belong to the animal and which are simply background. This task is known as semantic image segmentation.

Deep neural networks, it turns out, are very good at performing image segmentation. At Fritz, we’ve put in a lot of effort to train neural networks that are small enough and fast enough to segment images in a mobile app. Because everything runs on-device, it can be used in real-time and without internet connectivity, while also keeping user data private.

Set up your Fritz account and initialize the SDK

First, you’ll need a free Fritz account if you don’t already have one. Sign up here and follow these instructions to initialize and configure our SDK for your app.

If you don’t have an app but want to get started quickly, I’d recommend cloning our camera template app.

git clone https://github.com/fritzlabs/fritz-android-tutorials.git

To follow along, import CameraBoilerplateApp as a new project in Android Studio and then add the relevant code throughout this tutorial in the MainActivity.

Add the Pet Segmentation model via Gradle

In your app/build.gradle file, you can include the dependency with the following:

dependencies {
implementation ‘ai.fritz:core:3+’
implementation ‘ai.fritz:vision:3+’
implementation ‘ai.fritz:vision-pet-segmentation-model:3+’
}

This includes the pet segmentation model in the app. Under the hood, we use TensorFlow Lite as our mobile machine learning framework. In order to make sure that the model isn’t compressed when the APK is built, you’ll need to add the following in the same build file under the android option.

android {
    aaptOptions {
        noCompress “tflite”
    }
}

*You may also choose to lazy load the pet segmentation model to reduce your initial APK size. Follow the directions here under “Get a Segmentation Predictor.”

Create a Segmentation Predictor with a Pet Segmentation model

To run segmentation on an image, we’ve created a Predictor class that simplifies all the pre- and post-processing for running a model in your app. Create a new predictor with the following:

Run prediction on an image to detect pet

Images can come from a camera, a photo roll, or live video.

A good boy.

We’ll use this sample image as the input.

In the code below, we convert an android.media.Image object (YUV_420_888format) into a FritzVisionImage object to prepare it for prediction. This is usually the case when reading from a live camera capture session.

You may also convert a Bitmap to a FritzVisionImage.

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);

After you’ve create a FritzVisionImage object, call predictor.predict.

This will return a segmentResult that you can use to display the pet mask. For more details on the different access methods, take a look at the official documentation.

Use the mask to create a sticker

Now that we have the result from the model, let’s create a mask and use that to make a sticker of our pet. FritzVisionSegmentResult has a method called segmentResult.buildSingleClassMask that returns a mask for pets. The additional arguments are the max and min clipping values, respectively. Mask scores above the maximum value are clipped to 1 (totally opaque) while mask values below the min score are clipped to 0 (completely transparent).

Bitmap petMask= segmentResult.buildSingleClassMask(MaskType.PET, 255, .6f, .4f);

Using the mask, we can crop out the pixels from the original image.

Bitmap result = visionImage.mask(petMask)

The final image can be saved as a sticker!

Check out our GitHub repo for the finished implementation.

With Pet Segmentation, developers are able to create fun interactions between users and their animals. Simply add a couple of lines of code to create deeply engaging features that help distinguish your Android app from the rest.

Create a free Fritz account to get started. For additional resources, dive into the documentation or see a full demonstration in the open source Heartbeat app.