Creating a Pet Sticker App on iOS with Fritz Pet Segmentation
Turn dogs and cats into stickers to use in your favorite messaging apps
The more a phone’s camera understands, the more powerful and creative mobile applications can be. Few subjects are more important to understand than our pets: doggos, floofs, fluffers, clouds, woofers, and even cats (or so I’m told).
In this post, we’ll learn how to use Fritz, a mobile machine learning platform for iOS and Android, to give an iOS app the ability to generate pixel-perfect segmentation masks of pets in photos and video.
We’ll use this mask to create fun stickers you can share in apps like WhatsApp or Messenger, but you could also use the mask to track their growth over time, change the color of their fur, or perform occlusion in AR apps.
Defining the Task
There are many tasks we could teach our phone’s camera to perform: identifying a breed, locating an animal’s exact position, checking if a dog is sitting down, etc. In our case, we want to know exactly which pixels in an image belong to the animal and which are simply background. This task is known as semantic image segmentation.
Deep neural networks, it turns out, are very good at performing image segmentation. At Fritz, we’ve put in a lot of effort to train neural networks that are small enough and fast enough to segment images in a mobile app. Because everything runs on-device, it can be used in real-time and without internet connectivity, while also keeping user data private.
Pet Segmentation with Fritz
Download and open the repository for this tutorial.
It’s easiest to set up Fritz using Cocoapods. In the starter project folder, run:
pod repo update
Open the FritzPetStickerDemo.xcworkspace in Xcode.
All Fritz Vision APIs use a few constructs:
- FritzVisionImage: The image that the model runs on. It will wrap the provided pixel buffer or other images you provide.
- Options: Configuration options that are passed to the model letting you tweak how the model runs.
- Model: The actual model that runs predictions on the input images.
- Results: The output of the model. Each predictor has a different type of results. In this tutorial, the results are a list of
Set Up Fritz Account
- Create an account.
- Create an iOS App. Make sure that the Bundle ID of your project matches the one you created.
- Drag the
Fritz-Info.plistfile to your project.
After you run through the initialization steps, build and run your app. When your app has successfully checked in with Fritz, you’ll see a success message in the webapp.
Set Up an Image Picker
Next, we’re going to add the camera to our app. You can use Pet Segmentation on still images or video from the camera roll or live preview. For this tutorial, we’ll be choosing an image of a good boy from our camera roll.
This snippet opens an image picker and displaces the selected image on the background.
Create the Pet Predictor
Now it’s time to turn that selected image into a sticker.
Initialize the pet segmentation model as a variable in the view controller and run the model in the createSticker function:
Let's walk through this step-by-step:
- We start by setting some parameters that will be used to adjust the model output.
- We then create a
FritzVisionPetSegmentationModel()that will be used to cut out pets from an image.
- We update our
imagePickerControllerto call the
createStickerfunction with the image a user selects.
- That image is input into the segmentation model which returns a
FritzVisionSegmentationResultobject. For more details on the different access methods, take a look at the official documentation.
- The segmentation result contains information on the likelihood that each pixel belongs to a particular class (e.g. background or pet). We use the
buildSingleClassMaskmethod to create an alpha mask from just pixels that the model thinks are pets.
- That mask is combined with the original image using Core Image to produce a sticker with a transparent background using the
- That sticker is displayed to the user. You could also save that sticker for later user.
Here is a visual representation of what we’ve done:
Tweak mask sensitivity
Finally, we’ll take a look at the segmentation mask. With the prediction result, you can produce masks of varying sensitivity. The two important parameters are
zeroingScoresBelow. Confidence scores output by the model are between 0 and 1. All confidence scores above
clippingScoresAbove will go to 1. All confidence scores below
zeroingScoresBelow will go to zero. You can modify these parameters to create a soft edge.
Check out our GitHub repo for the finished implementation.
With Pet Segmentation, developers are able to create fun interactions between users and their animals. Simply add a couple of lines of code to create deeply engaging features that help distinguish your iOS app from the rest.