Having multiple people simultaneously seeing the same content in Augmented Reality is huge, and it’s now not only possible but very easily achievable with RealityKit.
Networking Permissions Update (iOS 14)
Since iOS 14, you need to specify that your app needs to be able to make advertising services and browse for other services on the network.
To do so, you must add
NSLocalNetworkUsageDescription, along with a short description to your Info.plist; similarly than is done for Camera Usage.
You then must also add NSBonjourServices. This creates an array of services which can be created by your app. The string(s) included in this array match the serviceType mentioned in the service initialisers, preceded with an underscore, and followed with
"._tcp". Below in the example we create a service named helper-test, so in the
Info.plist we should add
_helper-test._tcp. An example of this can be found in my repository example project here:
You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…
Intro to Browsing and Advertising MCSessions
The core functionality of connecting iOS devices starts at devices either browsing for hosts or advertising a session. For browsing you can either use the
MCNearbyServiceBrowser class, which uses a delegate to decide whether to connect to any given peer. Alternatively, you can use
MCBrowserViewController which displays a view where the user can choose from a list of peers.
For advertising a session you can use
MCNearbyServiceAdvertiser which also uses a delegate with pre-set rules to decide whether to let others connect to the host’s session. Alternatively, you can use
MCAdvertiserAssistant, which you can use to display UI pop-ups asking the host if they want to allow a user to join their session. Since I’m using this in the context of an Augmented Reality app, I would opt for the delegate options for both; otherwise we would break the AR experience. Here’s an example of how the advertiser functionality may look:
It is the browser’s job to look around for advertisers of sessions you can join and then inviting the advertiser to connect with them, and the advertiser handles those invitations to then connect the devices to the same MCSession. This connection works over wifi and bluetooth, optimizing itself the best way to do so in a similar way to Apple’s AirDrop protocol. Here’s a short example of how setting up the browser functionality might look:
From this point, the MCSession will manage the communication between all connected devices. Apple has provided a downloadable project which does exactly the above within the context of RealityKit. It uses an MCSession with RealityKit’s collaborative session API only. Therefore it does not use RealityKit’s synchronization, which is very straightforward to implement from there.
Creating a Collaborative Session
As an AR app runs, ARKit gathers information about a userʼs physical environment by processing the camera feed from the…
The collaborative session example above uses ARKit’s method session(_:didOutputCollaborationData:) to keep track of all the ARAnchors in the scene. The data is manually sent to all the peers, and when the peers receive it they use the update(with:) method to update the ARSession. The data in these examples is of a type ARSession.CollaborationData. The ARConfiguration’s
isCollaborationEnabled must be set to
true in order to send and receive these updates.
Something that RealityKit offers on top of this is MultipeerConnectivityService; this is a service which synchronizes all the entities within your scene graph along with their components.
Once you have an MCSession set up you just need to create a
MultipeerConnectivityService object and to tell the RealityKit scene to use that for its synchronizationService. Here’s an example of how that might look:
From here any AnchorEntity added to the scene will automatically inherit this property, and all Codable Components will also be transferred via the network.
If there is any entity which you do not want to be synchronized across the network with RealityKit automatically, you can set its
synchronization to nil. This will mean no updates from this entity or any of its child entities will be synchronized.
One important thing to note next is that every created entity has a designated owner. Only the owner of an entity can change that entity in such a way that the changes are distributed to all the other peers in the network; If someone is not already the owner of the entity they can simply request ownership of an entity, and the owner can make a request of an entity before changing any properties.
Here’s an example from a WWDC19 talk (linked below) on creating games with RealityKit, it shows how the host initially creates the two entities and sends them to the client. The host then makes an update to the top entity which is sent to the client. Shortly after the client tries to make an update to the other entity, but is denied because the client is not the owner of this entity. They then make a request to its owner (the host), it is accepted, and so the client is able to update the entity themselves as the new owner of the entity.
The base class for Entity uses a protocol called
HasSynchronization, this protocol contains a function for obtaining ownership of an entity,
requestOwnership. The example below is a similar setup to the request the client in the GIF above would send to the host.
After the ownership is has been transferred to the client, any changes that they now makes to the entity or any of its children entities will be reflected in the host’s scene, including transform, materials, and any other components it may have. If the host now makes any updates to
entity, they will have to request ownership in the same way.
There are two ways of granting ownership of an entity. The first is very simple, it just requires the entity’s
ownershipTransferMode to be switched to
.autoAccept. Doing so means that whenever an ownership request is made, the current owner will automatically grant the transfer.
Alternatively, if you want to keep the transfer mode in its default of
.manual, then you will need to use the
SynchronizationEvents.OwnershipRequest event. For more information on subscribing to events in RealityKit, please check out my previous article:
Getting started with RealityKit: Collisions and other Events
Part 4: subscribing to RealityKit Events
Someone may want to keep a transfer mode manual if the ownership depends on something that only the current owner’s device knows about the scene.
One final way to give ownership of an entity to another peer would be via the
giveOwnership method. This method works by just specifying the entity to transfer and the SynchronizationPeerID of the owner-to-be of the entity.
If the Entity ownership is still confusing, I’d highly recommend referring to Apple’s WWDC19 video which covers this very clearly, and can be found at around the 32:30 mark in the following video.
Building Apps with RealityKit - WWDC 2019 - Videos - Apple Developer
Gain a practical understanding of RealityKit capabilities by developing a game using its easy-to-learn API. Learn the…
I have created a class largely based off some examples from Apple, which aims to simplify creating shared experiences, both with and without RealityKit.
A light swift wrapper around the iOS MultipeerConnectivity framework. Including an example projectusing RealityKit’s MultipeerConnectivityService.
To initialise a peer-to-peer connectivity service with an iOS project and this small library all you’ll need to do is create an object of type
MultipeerHelper, setting the
.both and give it a
serviceName of a string following a few rules outlined here under
serviceType. Here’s an example:
self.multipeerHelp = MultipeerHelper(
This will create both a multi-peer advertiser and browser, this is great for when you don’t care about having a specific host user, which is a great way to get started with a collaborative session.
If you then want to use RealityKit’s
synchronizationService, then all you need to do is make sure that your configuration has the
isCollaborationEnabled parameter set to
true, and add the following line somewhere after
multipeerHelp is initialized:
self.arView.scene.synchronizationService = multipeerHelp.syncService
At this point all the devices need to do is let ARKit synchronise their ARAnchors by seeing the same positions from similar angles, as explained in the WWDC event here:
Building Collaborative AR Experiences - WWDC 2019 - Videos - Apple Developer
With iOS 13, ARKit and RealityKit enable apps to establish shared AR experiences faster and easier than ever…
From there, if you want to do more things over the network than just share entities, such as get notified when users join, leave, send data to each other etc. then you can use the
MultipeerHelperDelegate, from the
MultipeerHelper library to add callback functions. The full list of functions I have currently added to this delegate can be found here.
I made this small wrapper is because it seems that a lot of people don’t want to be creating delegates when creating shared AR experiences, they just want to get straight into it and concentrate on the Augmented Reality content. If you require something that this library does not yet offer, please feel free to open a Pull Request or, if you don’t know how to achieve it yourself, an Issue.
Thank you for reading, be sure to check out the example included in the
MultipeerHelper repository, I’ll be adding updates to that as I use collaborative AR more.
Here’s an AR game I have made using Collaborative Sessions. If you have questions about any part of it, leave a comment or send me a tweet!
If you learned something, or just enjoyed this article please tap the 👏 button as many times as you can. Any questions? Reply to this story or send me a tweet. If you want to see more follow me on Medium for future articles.