Audio/Video conferencing solution using XR

Remote collaboration in XR is getting popular these days, where users can collaborate in the space. Audio/Video conferencing in spatial collaboration is a kind of must-have feature. There are a number of conferencing services available traditionally for building business applications, few easy options are available for XR apps as well.

I am sharing my learning from implementing an audio conferencing solution using Unity. The best option I’ve found during my findings is the Agora.io Voice SDK for Unity available through the Unity Asset Store. In this article, I will explain how to integrate it into Unity. I also found other tools like VIVOX and Photon. I tried integrating Photon as well. But I found Agora.io much easy to use. Agora.io also provides 10,000 minutes for free which is quite helpful for a developer to start and test.

Let me tell you about Agora.io

Agora.io has there owned network infrastructure SD-RTN™, or Software Defined Real-time Network specifically designed for real-time communications in the world.

All audio and video services provided by the Agora SDK are deployed and transmitted through the Agora SD-RTN™. You can learn more about Agora.io architecture here.

Agora also provides built-in encryption as well as customized encryption. You can use either of them to implement encryption. The following diagram describes the encrypted data transmission process:

Pricing :

Agora provides the first 10,000 minutes free each month. Then you have to pay for its services you can check the pricing by clicking here. And for the billing process click here.

Prerequisites

• Unity Editor
• A developer account with Agora.io (see: How To Get Started with Agora)

Let’s Start

If you are new to Unity, please go through below article from Neelarghya —

Let’s get started with Unity…

To start, let’s open Unity and create a blank new project, I am giving the project name “XR Space”

The next step is to navigate to the Unity Store (if you are in the scene view, click the Unity Store tab) and search for “Agora voice SDK”.

Once the plugin page loaded, Go ahead and click on Download. After successful download click on Import to import the asset into your project.

Great, now we have imported the Agora package into our project.

Now we will create a simple scene named Login where we will add one User Name input field and one Space name input field and one button. I have also added a logo which is optional if you want you can add.

Now we create a script named XRSpaceManger where we will validate the user name and space name. Make sure all details are valid ( We are not going to manage user credentials here we only check input fields should not be null or empty).

using UnityEngine;
using UnityEngine.SceneManagement;
using UnityEngine.UI;

public class XRSpaceManager : MonoBehaviour
{
    private static XRSpaceManager m_Instance;
    public InputField mUserNameInputField;
    public InputField mSpaceNameInputField;
    public Button joinSpace;
    public Text mShownMessage;

    private void Start()
    {
        joinSpace.onClick.AddListener(JoinSpace);
    }
    
    private void JoinSpace()
    {
        if (string.IsNullOrEmpty(mUserNameInputField.text.Trim()))
        {
            mShownMessage.text = "Enter User Name";
            return;
        }

        if (string.IsNullOrEmpty(mSpaceNameInputField.text.Trim()))
        {
            mShownMessage.text = "Enter Space Name";
            return;
        }

        PlayerPrefs.SetString("userName", mUserNameInputField.text.Trim());
        PlayerPrefs.SetString("spaceName", mSpaceNameInputField.text.Trim());
        SceneManager.LoadScene("Space");
    }
}

Once we have user name and space name will store it in PlayerPrefs and the will load the Space scene. Which we are going to create in the next step. First, attach the XRSpaceManager script to the game object like this.

Let’s create a Space scene. In the apace scene we will add three major things 2 buttons one for mute and unmute and another one for leaving the space and for showing the currently active users in the space we will discuss in the next step.

Now let’s discuss showing active users. If we see in the above picture there is an “Active Users Holder” game object which nothing but an empty game object. In which we will add a vertical layout group and content size fitter for dynamic management of active user list.

Now we have UI let’s create Controller script name SpaceController.

using System.Collections;
using System.Collections.Generic;
using agora_gaming_rtc;
using UnityEngine;
using UnityEngine.Android;
using UnityEngine.SceneManagement;
using UnityEngine.UI;
using UnityEngine.UIElements;
using Button = UnityEngine.UI.Button;

public class SpaceController : MonoBehaviour
{
    public Text mShownMessage;
    public Text mSpaceName;
    public Button leaveChannel;
    public Button muteButton;

    [SerializeField] private Transform UserListHolder;
    [SerializeField] private GameObject userPrefab;
    [SerializeField] private string appId = "YOUR APP ID";

    private IRtcEngine mRtcEngine = null;
    private Dictionary<uint, User> _users = new Dictionary<uint, User>();

    void Awake()
    {
        QualitySettings.vSyncCount = 0;
        Application.targetFrameRate = 30;
        muteButton.enabled = false;
    }

    void Start()
    {
#if (UNITY_2018_3_OR_NEWER)
        if (Permission.HasUserAuthorizedPermission(Permission.Microphone))
        {
        }
        else
        {
            Permission.RequestUserPermission(Permission.Microphone);
        }
#endif
        muteButton.onClick.AddListener(MuteButtonTapped);
        leaveChannel.onClick.AddListener(LeaveChannelHandler);
        mRtcEngine = IRtcEngine.GetEngine(appId);
        mSpaceName.text = PlayerPrefs.GetString("spaceName");
        mRtcEngine.OnJoinChannelSuccess += JoinChannelSuccess;
        mRtcEngine.OnLeaveChannel += LeaveChannel;
        mRtcEngine.OnUserJoined += UserJoined;
        mRtcEngine.OnUserOffline += UserOffline;
        mRtcEngine.OnError += Error;
        JoinSpace();
    }

    private void JoinChannelSuccess(string channelname, uint uid, int elapsed)
    {
        muteButton.enabled = true;
    }

    private void Error(int error, string msg)
    {
        string description = IRtcEngine.GetErrorDescription(error);
        string errorMessage = string.Format("onError callback {0} {1} {2}", error, msg, description);
        Debug.Log(errorMessage);
    }

    private void UserOffline(uint uid, USER_OFFLINE_REASON reason)
    {
        if (_users.ContainsKey(uid))
        {
            _users[uid].gameObject.SetActive(false);
            _users.Remove(uid);
        }

        string userOfflineMessage = string.Format("onUserOffline callback uid {0} {1}", uid, reason);
        Debug.Log(userOfflineMessage);
    }

    private void UserJoined(uint uid, int elapsed)
    {
        UserInfo userInfo = mRtcEngine.GetUserInfoByUid(uid);
        User user = Instantiate(userPrefab, UserListHolder).GetComponent<User>();
        user.Initialize(userInfo.userAccount, false);
        _users.Add(uid, user);
        string userJoinedMessage = string.Format("onUserJoined callback uid {0} {1}", uid, elapsed);
        Debug.Log(userJoinedMessage);
    }

    private void LeaveChannel(RtcStats stats)
    {
        muteButton.enabled = false;
        if (isMuted)
        {
            MuteButtonTapped();
        }
    }


    public void JoinSpace()
    {
        string spaceName = mSpaceName.text.Trim();

        Debug.Log(string.Format("tap join with channel name {0}", spaceName));

        if (string.IsNullOrEmpty(spaceName))
        {
            return;
        }
        mRtcEngine.RegisterLocalUserAccount(appId, PlayerPrefs.GetString("userName"));
        mRtcEngine.JoinChannelWithUserAccount(null, spaceName, PlayerPrefs.GetString("userName"));
    }

    bool isMuted = false;

    void MuteButtonTapped()
    {
        string labeltext = isMuted ? "Mute" : "Unmute";
        Text label = muteButton.GetComponentInChildren<Text>();
        if (label != null)
        {
            label.text = labeltext;
        }

        isMuted = !isMuted;
        mRtcEngine.EnableLocalAudio(!isMuted);
    }

    public void LeaveChannelHandler()
    {
        StartCoroutine(LoadSceneCoroutine());
    }

    private void OnDestroy()
    {
        mRtcEngine.OnLeaveChannel -= LeaveChannel;
        mRtcEngine.OnError -= Error;
        mRtcEngine.OnUserJoined -= UserJoined;
        mRtcEngine.OnUserOffline -= UserOffline;
    }

    private IEnumerator LoadSceneCoroutine()
    {
        yield return new WaitForSeconds(0.5f);
        SceneManager.LoadScene("Login");
    }
}

Attach this script to the Space controller game object.

If we see there is an App Id field that is needed for SDK to make connections to the server. At this point, we need to log into our Agora.io developer account and create a new AppID or select an existing one. I had previously created so I will use this.

Copy the App Id and paste it in the SpaceController App Id field.

Now, all set you can build this scene or you can also run it in the editor.
It will look like this. Just enter the user name and channel name hit join Space. Build and export it as an android app.

Demo

Note: You can build it for any platform Android, iOs or Windows. For testing voice chat functionalities you have to deploy the app into two devices. When both the devices connect to same channel then you will be able to speak and hear audio on each device.

Conclusion

We have seen a basic working of Agora.io voice chat in unity. Now we can integrate it easily to any of the unity based applications wherever we need audio conferencing.