As augmented reality becomes more mainstream, we will see new emerging use-cases that involve AR and real-time collaboration and sharing. To help iOS developers, I created a framework that combines Agora.io’s Video SDK and ARKit. This makes it easier to add real-time collaboration to their AR apps.
ARKit uses the device’s camera and motion sensors to project virtual content into a user’s world. Agora.io provides a video SDK for building real-time video and audio communications applications.
Overview
Before we dive into how to use AgoraARKit, let’s take a look at the framework structure, and dependencies (device/OS). AgoraARKit contains 4 classes with managed user interfaces:
Lobby: the pre-channel UIView, provides a text input for users to define their channel name and choose their role (broadcaster and audience)ARBroadcaster: User broadcasting their AR view in the live streamARAudience: User viewing the remote user’s AR session.Broadcaster: With a device that does not support the ARConfiguration but needs to be able to broadcast their video stream.
Device Requirements
The ARBroadcaster users inherit device limitations from ARKit, while the device requirements are lower for the ARAudience users. For a user to broadcast their AR content into a channel they must be and using iOS 12 or later on:
- iPhone 6S or newer
- iPhone SE
- iPad (2017)
- All iPad Pro models
Framework Dependencies
AgoraARKit uses Apple’s ARKit framework but also relies on the Agora.io Video SDK and ARVideoKit. I could have built my own off-screen renderer, but given ARVideoKit’s popularity and its ability to record a variety of formats, I felt it was a better solution.
Note: Sign up here if you don’t have an Agora developer account.
Demo App
Let’s quickly get started with a demo app. Start with a new single view project. I named mine AgoraARKit Demo and for this example, we’ll stick to the Storyboard interface.
Add the framework
Next let’s add the AgoraARKit framework to our project. Create a Podfile , open it and add the AgoraARKit pod.
Podfle for referencePermissions
Add NSCameraUsageDescription, NSMicrophoneUsageDescription, NSPhotoLibraryAddUsageDescription, and NSPhotoLibraryUsageDescription to the info.plist with a brief description for each. The last two permissions are required by ARVideoKit because of its ability to store photos/videos.
Note: I am not implementing on-device recording so we don’t technically need any of the library permissions; but if you plan to use this in production you will need to include them because they are requirements for ARVideoKit. For more information review Apple’s guidelines on permissions.
Remove Scene Delegate
Since we are using the Storyboard interface, we can remove the SceneDelegate.swift and the Scene Manifest entry from the info.plist. Then we need to open the AppDelegate.swift and remove the Scene Delegate methods, and add the window property. Your AppDelegate.swift should look like this:
Implementation
Open your ViewController.swift, add import AgoraARKit just below the import UIKit line and set your ViewController class to inherit from AgoraLobbyVC. Next set your Agora App Id within the loadViewmethod. Let’s also set a custom image for the bannerImage property.
Customizations
You’re done with the basic implementation! Yeah, it was that simple. Now let’s take a quick look at how we can customize the AgoraARKit classes.
LobbyVC
Since we are already inheriting from the AgoraLobbyVC, let's override the joinSession and createSessionmethods within our ViewController to set the images for the audience and broadcaster views.
ARBroadcaster
The ARBroadcaster is a UIViewController that implements the ARKit Session and Render Delegates along with the Agora RTC Engine Delegate methods. For a full list of each, please see the AgoraARKit documentation.
The current ARBroadcaster class is set up for WorldTracking, but this can be easily updated to front-facing. Below is an example of the ARBroadcaster extended for ARKit FaceTracking and also adds support for multiple broadcasters.
Done!
Wow, that was quick! Thanks for following along and please leave a comment to let me know what projects you are working on using AgoraARKit!
