Agora.io
Published in

Agora.io

Create a SwiftUI Streaming App with Fun Virtual Backgrounds

Creating a video streaming app is super straightforward with the Agora SDK and Agora UIKit. But what about adding cool features to enhance the overall user experience or adding monetisation options? That’s where the Agora Extensions Marketplace comes in.

Launched in September 2021, the Agora Extensions Marketplace is a one-stop-shop for app enrichment features like background and foreground segments, voice changers, and much more.

In this post, we’ll show you how to add or change eye-catching backgrounds to your streaming app with the VisionLab DoreSegment extension. With the DoreSegment extension, you can get rid of background noise in three easy steps:

  1. Enable the extension.
  2. Pass extension credentials.
  3. Pass an image as the new background.

Now let’s get started

Prerequisites

Setup

Let’s start with a new, single-view iOS project. Create the project in Xcode, choosing SwiftUI for the interface, and then add the Agora UIKit package.

Add the package by selecting File > Swift Packages > Add Package Dependency, and then paste the link into this Swift package:

https://github.com/AgoraIO-Community/iOS-UIKit.git

When asked for the version, insert 4.0.0-preview, and select "Up to Next Major". This should install at least version 4.0.0-preview.8, which is released at the time of writing.

When in doubt, you can always select “Exact” and insert 4.0.0-preview.8.

Once that package is installed, the camera and microphone usage descriptions need to be added. To see how to do that, check out Apple’s documentation here:

Video Call UI

I won’t go into much detail about the UI here, because we’re mainly focusing on the background-replacing extension, but on a high level:

  • The ContentView contains two views: an AgoraViewer (defined in Agora UIKit) and a button to join the channel.
  • When the join button is pressed, the Agora UIKit method join(channel:,with:,as:) is called, which joins the Agora video channel.

This is what the ContentView looks like:

The AgoraViewerHelper helper object, without any of the DoreSegment additions looks like this:

In the above snippet, we have added a button with a magic wand image. When clicked, it will call the method, toggleBackground.

We now have a working video call app with SwiftUI and Agora UIKit. The next step is to integrate the DoreSegment extension!

Integrating DoreSegment

Credentials

If you have an account with Agora and are currently signed in, follow this link to activate the extension for your account.

First, you will need to activate DoreSegment for your account by clicking “Activate” here:

Next, enable DoreSegment for the project, the same project as for the Agora App ID used in the app:

Once that’s done, you can grab your DoreSegment API Key and API Secret by clicking “view” below the Credentials column:

Add to Xcode

We need to add the extension, which can also be installed via the Swift Package Manager with the following URL:

https://github.com/AgoraIO-Community/Extension-VisionLab-DoreSegment-iOS.git

The latest release at the time of writing is 1.0.1.

This package doesn’t need importing into your Swift code. It just needs to be bundled alongside the Agora SDK in your app target.

Enable the Extension

When using Agora extensions, you need to enable them before joining a channel.

To do this with Agora UIKit, you can call AgoraVideoViewer.enableExtension().

This method needs to be passed the vendor (“DoreAI”) and the vendor extension (“DoreSegment”) in this case:

AgoraViewerHelper.agview.viewer.enableExtension(
withVendor: "DoreAI", extension: "DoreSegment", enabled: true
)

The above code snippet is added to the beginning of the joinChannel method defined earlier.

After joining the channel, we are going to disable the extension again because we want the feature to be disabled until someone clicks the button to set a new background. But the feature must be enabled once before joining the channel, so we must enable and then disable it:

func joinedChannel(channel: String) {
registerDoreSegment()

AgoraViewerHelper.agview.viewer.enableExtension(
withVendor: "DoreAI", extension: "DoreSegment", enabled: false
)
}

Toggle the Extension

As shown in the above snippet for AgoraViewerHelper, we have added an extra button to the default buttons provided by Agora UIKit.

This button displays the magic wand icon and toggles between a green and a grey background. When clicked, the button calls a method toggleBackground. Let’s link up this button to enable and disable the extension, and apply an image as the background image (bgImage):

@objc func toggleBackground(_ sender: UIButton) {
virtualBackgroundEnabled.toggle()
sender.backgroundColor = self.virtualBackgroundEnabled ?
.systemGreen : .systemGray
if self.virtualBackgroundEnabled {
let backgroundImg = UIImage(named: "background-boat")
guard let base64Img = backgroundImg?.pngData()
?.base64EncodedString()
else {
fatalError("Background image not found in bundle")
}
AgoraViewerHelper.agview.viewer.setExtensionProperty(
"DoreAI", extension: "DoreSegment",
key: "bgImage", value: base64Img
)
}
AgoraViewerHelper.agview.viewer.enableExtension(
withVendor: "DoreAI", extension: "DoreSegment",
enabled: self.virtualBackgroundEnabled
)
}

In the above snippet, we fetch an image from the app's bundle named background-boat if the background is now enabled. That image is passed through to DoreSegment so that it knows what image we want as the background. Then we call enableExtension again, either enabling or disabling the extension. The image used in the background is included in the GitHub project but can also be downloaded here.

Almost there now! The only thing left to do is define the registerDoreSegment method.

This method needs to pass the API Key and API Secret values. These values need to be passed to the extension with the keys apiKey and license respectively:

func registerDoreSegment() {
// Set API Credentials
let doreCredentials: [String: String] = [
"apiKey": AppKeys.visionLabApiKey,
"license": AppKeys.visionLabApiSecret
]
AgoraViewerHelper.agview.viewer.setExtensionProperty(
"DoreAI", extension: "DoreSegment",
key: "start", codable: doreCredentials
)
}

This is the final result:

Conclusion

Congratulations! You now have a video call app complete with the new extension from VisionLab!

There are other extensions I’d encourage you to try out. They can all be found here:

If you want to create an app from scratch with this extension, the extension can be applied to an application made directly with the Agora SDK. For that, you will need to make sure you’re using the 4.x version of the SDK. Details can be found here:

The methods, such as setExtensionProperty, used in this post are adaptations of the SDK's built-in methods relating to extensions. They can be found in this file from Agora Video UIKit:

Testing

You can try out this app by following this GitHub link:

Other Resources

For more information about building applications using Agora SDKs, take a look at the Agora Video Call Quickstart Guide and Agora API Reference.

I also invite you to join the Agora Developer Slack community to meet with our developer team as well as other like-minded developers and technology enthusiasts.

--

--

--

Agora delivers cutting edge APIs for developers. Our blog covers the world of technology.

Recommended from Medium

⭐️Vacante del día⭐️

Widgets in iOS

WP Swift Administration PRO v1.4.7 — IndirPremium

WP Swift Administration PRO v1.4.7 – IndirPremium

How to implement sound waves in iOS by using ZEGOCLOUD SDK

sound waves

Xamarin Tips: Check if notification permission got granted for iOS in Xamarin.Forms

Customizing Buttons in SwiftUI iOS 15

graphic showing title of the article and Swift logo

Swift Programming Tutorial: Functions

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Max Cobb

Max Cobb

Excited about all things Augmented Reality! Developer Evangelist at Agora.io.

More from Medium

Augmented Performance in SwiftUI & RealityKit

Classifying Sounds with the SoundAnalysis framework and custom CoreML models in Swift

Universal Links and SwiftUI Video Calls

Build an iOS framework for distribution with Bazel