Turn dogs and cats into stickers to use in your favorite messaging apps

Jameson Toole
Jun 20 · 5 min read
Image Source

The more a phone’s camera understands, the more powerful and creative mobile applications can be. Few subjects are more important to understand than our pets: doggos, floofs, fluffers, clouds, woofers, and even cats (or so I’m told).

In this post, we’ll learn how to use Fritz, a mobile machine learning platform for iOS and Android, to give an iOS app the ability to generate pixel-perfect segmentation masks of pets in photos and video.

We’ll use this mask to create fun stickers you can share in apps like WhatsApp or Messenger, but you could also use the mask to track their growth over time, change the color of their fur, or perform occlusion in AR apps.


Defining the Task

There are many tasks we could teach our phone’s camera to perform: identifying a breed, locating an animal’s exact position, checking if a dog is sitting down, etc. In our case, we want to know exactly which pixels in an image belong to the animal and which are simply background. This task is known as semantic image segmentation.

Deep neural networks, it turns out, are very good at performing image segmentation. At Fritz, we’ve put in a lot of effort to train neural networks that are small enough and fast enough to segment images in a mobile app. Because everything runs on-device, it can be used in real-time and without internet connectivity, while also keeping user data private.


Pet Segmentation with Fritz

Starting out

Download and open the repository for this tutorial.

It’s easiest to set up Fritz using Cocoapods. In the starter project folder, run:

pod repo update
pod install

Open the FritzPetStickerDemo.xcworkspace in Xcode.

Overview

We’ll be using the Fritz iOS Image Segmentation feature to generate masks for pets in photos. The Fritz SDK comes with a variety of pre-built features that run directly on your phone.

All Fritz Vision APIs use a few constructs:

  • FritzVisionImage: The image that the model runs on. It will wrap the provided pixel buffer or other images you provide.
  • Options: Configuration options that are passed to the model letting you tweak how the model runs.
  • Model: The actual model that runs predictions on the input images.
  • Results: The output of the model. Each predictor has a different type of results. In this tutorial, the results are a list of FritzVisionSegmentationResult objects.

Set Up Fritz Account

Setting up a Fritz Account is easy. Follow the Getting Started directions to set up your Fritz Account and connect the demo to your account. Here are the steps you’ll run through:

  1. Create an account.
  2. Create an iOS App. Make sure that the Bundle ID of your project matches the one you created.
  3. Drag the Fritz-Info.plist file to your project.

After you run through the initialization steps, build and run your app. When your app has successfully checked in with Fritz, you’ll see a success message in the webapp.

Set Up an Image Picker

Next, we’re going to add the camera to our app. You can use Pet Segmentation on still images or video from the camera roll or live preview. For this tutorial, we’ll be choosing an image of a good boy from our camera roll.

This snippet opens an image picker and displaces the selected image on the background.


Create the Pet Predictor

A good boy

Now it’s time to turn that selected image into a sticker.

Initialize the pet segmentation model as a variable in the view controller and run the model in the createSticker function:

Let's walk through this step-by-step:

  1. We start by setting some parameters that will be used to adjust the model output.
  2. We then create a FritzVisionPetSegmentationModel() that will be used to cut out pets from an image.
  3. We update our imagePickerController to call the createSticker function with the image a user selects.
  4. That image is input into the segmentation model which returns a FritzVisionSegmentationResult object. For more details on the different access methods, take a look at the official documentation.
  5. The segmentation result contains information on the likelihood that each pixel belongs to a particular class (e.g. background or pet). We use the buildSingleClassMask method to create an alpha mask from just pixels that the model thinks are pets.
  6. That mask is combined with the original image using Core Image to produce a sticker with a transparent background using the masked method.
  7. That sticker is displayed to the user. You could also save that sticker for later user.
A pastable sticker for your messaging apps.

Here is a visual representation of what we’ve done:

The original image (left). The alpha mask of pixels containing our pet, purple denotes background, bright yellow, a pet (middle). The final sticker (right).

Tweak mask sensitivity

Finally, we’ll take a look at the segmentation mask. With the prediction result, you can produce masks of varying sensitivity. The two important parameters are clippingScoresAbove and zeroingScoresBelow. Confidence scores output by the model are between 0 and 1. All confidence scores above clippingScoresAbove will go to 1. All confidence scores below zeroingScoresBelow will go to zero. You can modify these parameters to create a soft edge.

Check out our GitHub repo for the finished implementation.


With Pet Segmentation, developers are able to create fun interactions between users and their animals. Simply add a couple of lines of code to create deeply engaging features that help distinguish your iOS app from the rest.

Create a free Fritz account to get started. For additional resources, dive into the documentation or see a full demonstration in the open source Heartbeat app.

Better Programming

Advice for programmers.

Thanks to Zack Shapiro

Jameson Toole

Written by

Michigan. MIT. Data.

Better Programming

Advice for programmers.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade