Using Sky Segmentation to create stunning background animations in iOS

Eric Hsiao
Jun 24, 2019 · 4 min read
Image for post
Image for post

Image segmentation is a task that labels each pixel in an image with the type of object it belongs to. In short, it helps us better understand images / video so that we can build engaging experiences that users love.

In a , we explored how developers can use People Segmentation in order to create a portrait mode effect by separating the background from people detected in an image and then blurring it. Here, we’ll work with another flavor of on-device image segmentation: Sky Segmentation.

Sky Segmentation works the same way as People Segmentation, using an on-device machine learning model to allow developers to associate pixels of an image or video frame with the sky. For this tutorial, we’ll work through the steps to create a sky animation using by Fritz.

Image for post
Image for post

Starting out

and open the repository for this tutorial.

git clone :fritzlabs/fritz-examples.git

It’s easiest to set up Fritz using Cocoapods. In starter project folder, run:

pod repo update
pod install

Open the in Xcode.

Overview

We’ll be using the feature to generate masks for pets in photos. The Fritz SDK comes with a variety of that run directly on your phone.

All Fritz Vision APIs use a few constructs:

  • FritzVisionImage: The image that the model runs on. It will wrap the provided pixel buffer or another image you provide.
  • Options: Configuration options that are passed to the model letting you tweak how the model runs.
  • Model: The actual model that runs predictions on the input images.
  • Results: The output of the model. Each predictor has a different type of results. In this tutorial, the results are a list of FritzVisionSegmentationResult objects.

Setup Fritz Account

is easy. Follow the directions to setup your free account and connect the demo to your account. Here are the steps you’ll run through:

  1. Create a free account for developers.
  2. Create an iOS App. Make sure that the Bundle ID of your project matches the one you created.
  3. Drag the Fritz-Info.plist file to your project.

After you run through the initialization steps, build and run your app. When your app has successfully checked in with Fritz, you’ll see a success message in the webapp.

Using Sky Segmentation

Let’s use sky segmentation on an image and then replace the sky pixels with a new, sliding background.

Image for post
Image for post
Left: Original Image | Top Middle: Sky photo to replace | Top Bottom: Original image without the sky | Right: Combine background sky with the bitmap.

In our app, first let’s load 2 images:

  • A foreground image that we’ll run sky segmentation on.
  • A background image that we’ll use to animate across the sky.
Image for post
Image for post
Left: foreground image (mountains.jpg) | Right: background image (clouds..png)
let foreground = UIImage(named: "mountains.jpg")
let background = UIImage(named: "clouds.png")

Now we’re ready to pass the image into the model. Let’s walk through this step-by-step.

  1. We start by setting some parameters that will be used to adjust the model output.
In step 3, we’ll use these parameters to adjust the model ouput when the buildSingleClassMask method is called.

2. We then create a FritzVisionSkySegmentationModel() that will be used to cut out the sky from the image.

private lazy var visionModel = FritzVisionSkySegmentationModel()

3. The foreground image is fed into the segmentation model which returns a FritzVisionSegmentationResult object. For more details on the different access methods, take a look at the .

The segmentation result contains information on the likelihood that each pixel belongs to a particular class (e.g. background or sky). We use the buildSingleClassMask method to create an alpha mask from just pixels that the model thinks does not identify as the sky.

4. That mask is combined with the original image to produce an image with a transparent sky.

5. Add the background UIImage loaded. In this case, we’ll use 2 background views and then animate them by creating a sliding motion that repeats indefinitely.

Here’s the final result:

Image for post
Image for post

Check out our for the finished code.


By being able to easily identify the sky, we open up the door for developers to create novel features.

  • Animate the background sky — add animated stars to the sky to give it some pop; create a moving background that brings the photo to life.
  • Sky replacement — Compose and edit the background of videos and photos. Want to create a movie poster for Godzilla? Find a photo of a city skyline and swap out the background.
  • Photo effects targeting the sky — Add filters specifically for the sky without the need to apply it to the entire background. Add a reddish glow to the sky as the sun comes up in the morning.

When working with video and photo apps, you can add intelligent, computer vision experiences — all without a PhD in machine learning. I hope that this tutorial showed how easy it is for any developer to use Image Segmentation by Fritz. Leave a comment and let us know what you build!

Flawless iOS

🍏 Community around iOS development, mobile design, and…

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store