AI POWERED XR SERIES

Meta AI : Developing AI powered XR App for Meta Quest

In this series of articles, we will explore how Meta AI is powering the next generation of XR experiences, and in this article let’s learn about developing XR application using Meta AI for Meta Quest.

⬅ Previous|| Current |️|️ Next ➡

In the previous article, we introduced you to Meta AI and it’s usecases in (eXXR. In this one, we will provide a step-by-step tutorial on building an XR app using Meta AI for the Meta Quest headset. Let’s dive in and explore this exciting technology further.

Meta AI: Powering Next Generation of XR experiences

Meta Quest XR Device

The Meta Quest is an advanced extended reality (XR) device developed by Meta, designed to immerse users in 3D spaces. The latest models, Quest 3 and Quest Pro, feature a full-color passthrough mode, a significant upgrade from the black-and-white passthrough of the Quest 2. This technology allows users to blend virtual elements seamlessly with their physical surroundings, creating what is known as mixed reality (MR). By merging virtual objects with real-world environments, the Meta Quest enhances immersion, making the interaction with virtual elements feel more tangible and lifelike.

Picture taken at Thoughtworks office while working on Meta Quest 3

Application of XR technology

Before getting into XR development, let’s quickly cover various applications in the domain.

• Virtual Training in Various Fields — In healthcare, XR technology can be used for virtual training simulations that allow medical professionals to practice procedures in a realistic environment without risk to patients. Similarly, it can also be used for training in aviation through flight simulations
• Entertainment — In the entertainment sector, XR transforms gaming experiences by immersing players in virtual worlds where they can interact with other characters and environments. People can also attend any social events from anywhere in the world through VR, providing a sense of presence and interaction that traditional broadcasts cannot replicate.
• Retail — Retailers leverage XR for virtual shopping experiences that simulate physical stores or showcase products in immersive environments. Customers can preview furniture placements in their homes through AR apps or try on virtual clothing before making purchase decisions, enhancing the online shopping experience. More details here.
• Open Space and Museums, Exhibition may well use XR technology to enhance the visitor engagement, by creating more immersive and creating experiences. Ref more here

More use cases on eXtended Reality described by Thoughtworks articles and also explore 100s of use-cases of metaverse with XR, AI discussed by in my colleague Kuldeep Singh, in his book Exploring the Metaverse. It is now available globally

📕 Exploring the Metaverse | Available Globally

Building XR Apps for Meta Quest

Unity 3D is a powerful development environment for building XR apps for multiple devices include apps for Meta Quest. My colleague Neelarghya’s article may help you in getting started with Unity, if you are very new to it, else you may skip this section.

Let’s get Started with Unity…

Building XR Apps using Meta AI for Quest

Assuming you already know Unity3D, here we will jump on MetaAI based XR app development for Quest. Here are the easy steps to utilize Meta AI:

Create an Account and Generate API Key for AI Inference:

Visit Groq Playground, a AI Inference Engine that provide various AI access though APIs, we need to create an an account and API keys there.

Generate an API key.

Create a Unity 3D project and Setup for Quest

Go to Unity Hub and create a new 3D project

Add Quest Related Plugins:

• Navigate to the Edit → project settings → Install XR plugin management settings and enable Oculus

• Download Meta All-in-One SDK from the asset store and import to your Unity project by going to Window → package manager → My assets → select meta All-in-one SDK → click on the install button

• Go to Edit → project settings → player → other settings → Enable download over HTTP for development builds

• Check the developer build option for your app in the build settings.

Add OVR Camera Rig Prefab:

• In the project window search for OVR Camera rig, and drag and drop it to the scene hierarchy

• In the Unity inspector, go to the OVR Manager component and enable the Quest devices for which you are building the app.

• Make sure that under Quest Features, requires system keyboard is checked for the OVR camera.

Add OVR Interaction Comprehensive Prefab:

In the project window search for OVR Interaction Comprehensive Prefab, and drag and drop it to the scene hierarchy

• Navigate to OVR Controllers → Left Controller → OVR Controller Visual.
• Disable the game objects of controller prefabs which are not relevant. Repeat this step for the right controller.

Create a User Interface for Quest

• Download Meta XR Interaction SDK OVR Samples and import them into the project.
• Follow the tutorial to create a basic UI with buttons and input fields.

• After creating the canvas by following the tutorial above, go to canvas game object → Canvas component → Event camera. Drag and drop the ovr Camera Rig → Tracking Space Centre Eye Anchor in the field

Add Meta Voice SDK

• Integrate Meta’s Voice SDK to enable speech-to-text and text-to-speech functionalities. Tutorial for voice SDK integration:
  Speech to Text: link
  Text to Speech: link

MetaAI Integration

Create a script named GroqApiClient to interact with Meta AI’s Llama 3 model through Groq by adding the following code:

public class GroqApiClient{
private readonly HttpClient client;
  public GroqApiClient(string apiKey){
    client = new HttpClient();
    client.DefaultRequestHeaders.Add("Authorization", $"Bearer {apiKey}");
  }
  public async Task<string> CreateChatCompletionAsync(string prompt){
    var httpContent = new StringContent(
    "{\"messages\": [{\"role\": \"user\",\"content\": \"" + prompt + "\"}],\"model\": \"llama3–8b-8192\"}",
    Encoding.UTF8,
    "application/json"
    );
    using var response = await client.PostAsync("https://api.groq.com/openai/v1/chat/completions", httpContent);
    response.EnsureSuccessStatusCode();
    var body = await response.Content.ReadAsStringAsync();
    var data = JsonConvert.DeserializeObject<Response>(body);
    Debug.Log(body);
    return data.choices[0].message.content;
  }
}
public class Response{
  public Choice[] choices;
}
public class Choice{
  public Message message;
}
public class Message{
  public string role;
  public string content;
}

Create game object and script to interact with the llama3 model:

• Go to the scene hierarchy and create an empty game object name it llamaController
• Create a script with the same name and a method to send requests to the llama 3 model through its API, extract the details, and provide it to the user. Attach it to the game object created in the previous step

using System.Text.RegularExpressions;
using System.Threading.Tasks;
using GroqApiLibrary;
using Meta.WitAi.Dictation;
using Meta.WitAi.TTS.Utilities;
using TMPro;
using UnityEngine;
using UnityEngine.UI;

public class LlamaController : MonoBehaviour
{
    private const string API_KEY = "your api key";

    private GroqApiClient _groqApiClient ;
    private string request ;
    
    [SerializeField] private TMP_Text outputField;
    [SerializeField] private Toggle _toggle;
    [SerializeField] private TMP_Text message;
    [SerializeField] private TMP_InputField inputField;
    [SerializeField] private TTSSpeaker ttsSpeaker;
    [SerializeField] private DictationService _dictation;

    void Start()
    {
        _groqApiClient = new GroqApiClient(API_KEY);
    }

    public void ButtonOnclick()
    {
        SendChatCompletionRequest();
    }

    public async Task SendChatCompletionRequest()
    {
        if (_toggle.isOn)
          request = message.text;
        else
            request = inputField.text;
        request = Regex.Replace(request, @"\s+", " ");
        string result = await _groqApiClient.CreateChatCompletionAsync(request);
        outputField.text = result;
        PlayAudio(result);
    }
    
    public void ClearText()
    {
        message.text = "";
    }
    
    private void PlayAudio(string result)
    {
        if (_dictation.MicActive) 
            _dictation.Deactivate();
        ttsSpeaker.Speak(result);
    }
}

Run the XR app using MetaAI in Quest

Build the project and deploy it in Quest device, and enjoy your first Meta AI based XR app.

Here is a Demo of the app.

Conclusion

With this, we complete a basic tutorial for building an AI-powered XR app using Meta AI. This is a very simple app with minimal features, and you might wonder what value such a 2D canvas adds to XR. You are right, many be nothing! However, this foundation can be expanded significantly. Imagine a photorealistic virtual avatar interacting with you in a virtual or real-world MR mode, creating a more realistic use case. Similarly, incorporating multimodal AI responses for spatial environments and recognizing surroundings and people could elevate the experience to the next level.

In the next article, we will explore this example on other devices. Stay tuned.

⬅ Previous|| Current |️|️ Next ➡