Rendering immersive web experiences with Three.JS and WebXR

9 min readMar 1, 2019

This article is part of a series about creating responsive VR experiences:

Part 1: VR concepts and the immersive web
Part 2: Rendering immersive web experiences with Three.JS and WebXR (this article)
Part 3: Adding support for VR inputs with WebXR and Three.JS

In the first post, we learned about the VR concepts and how they are mapped in WebXR. This allows you to think about the experience you want to provide to your user. In this article, we’re going to cover how you can use WebXR together with Three.JS to create immersive experiences that target a large and heterogeneous user base.

A FEW WARNING WORDS

The WebXR API is still being refined (the First Public Working Draft has just been published) so I will try my best to update this series to reflect the changes and keep these articles ever-green. New features will come to the WebXR Device API specification and I will update this article accordingly if it simplifies the code.

QUICK OVERVIEW OF THREE.JS RENDERING PIPELINE

I’m not going to spend much time talking about how the Three.JS rendering pipeline works, because it is well documented on the internet (for example here). I’m just going to lay down the basics in the following diagram so it’s easier to understand where the pieces go.

Very basic overview of rendering with Three.JS

GETTING STARTED WITH THE WEBXR DEVICE API

Before we get into the WebXR API itself, you should be aware that the WebXR Device API Polyfill is available to help developers in two main ways:

If the device/browser doesn’t support the WebXR Device API, it will try to polyfill it using the available sensors such as gyroscope and accelerometer, allowing developers to provide a basic cardboard-like experience or inline rendering.
If the browser supports the legacy WebVR API, it will polyfill the WebXR Device API on top of WebVR, allowing developers to leverage all the work done to support WebVR in the first place (and thus allowing it to leverage the VR runtime underneath it).

The main types of 3D experiences that users can move into include:

Keyboard/mouse based on a desktop computer with no immersive support whatsoever.
Inline rendering or magic window leveraging the sensors of the phone. Inline rendering is a great way to “tease” users about your content, showing them a glimpse of your experience in the hopes that it will make them click the button and enter the more immersive experience inside the HMD. Here is an example:

Immersive VR with a dedicated VR system, mobile based VR or cardboard like experiences.

The WebXR Device API is based on the concept of sessions. You request a session and the browser takes care of starting the rendering in the HMD, for example. When you end the session, the rendering stops inside the HMD and you can start rendering again as usual on your page. There are 3 types of sessions: immersive-vr, immersive-ar (not covered in this article), and inline.

Here is a very simple flow chart that can help you decide which experience to provide, depending on factors such as capabilities of the device or support of the WebXR Device API.

REQUESTING A SESSION AND RENDERING CONTENT

This section describes the high-level flow of the required steps to request a session and render content with the WebXR Device API. We’ll go into more details in some of the steps, focusing mostly on the progressive aspect rather than the rendering itself.

During this article we’re going to refer to a simple demo located here. It uses WebXR and Three.JS and is the basis of the code snippets in this article. The full source is located here.

CHECK WEBXR SUPPORT

It’s not uncommon that you will find browsers or devices that do not support the WebXR Device API (even with the WebXR polyfill). In this case, rather than serving an empty page, you can consider a fallback based on keyboard and mouse, so the user can navigate inside the experience (similar to a 3D game).

In Three.JS it’s relatively straightforward to support this. You can use the PointerLockControls class to automatically map the mouse to move the camera (in the same way as a FPS shooter). The benefits of using pointer locking is that when the lock is acquired, it will send deltas of mouse movements rather than their absolute positions in the viewport. An additional benefit is that the cursor can’t go out of the browser window unless the user unlocks the pointer (typically using the escape key, giving you a way to pause the experience), and the cursor becomes hidden. This is ideal for what we need.

Note that THREE.PointerLockControls will not lock the pointer for you. Typically, you want to interact with a button to get started, which alerts the user that something will happen. Here is a piece of simplified code doing this:

THREE.PointerLockControls will take care of updating the camera whenever the mouse is moved.

The last part to handle is keyboard movements, which is pretty straightforward:

In your handlers you can just update the position of the camera that you store in some variable. In the demo, I store the intended direction of movement, because I smooth out the movement by having a velocity so it doesn’t look like the user is jumping positions.

When you’re done updating the position inside your render function, then update THREE.PointerLockControls:

Then render your scene as usual and loop again with requestAnimationFrame:

IF WEBXR IS SUPPORTED, CHECK THE SUPPORTED SESSION MODES

If WebXR is supported in your browser (either natively or with the polyfill), you want to query the supported modes for a XR session to decide next steps, for example, to add a button to enter into an immersive mode. Here is an example trying to figure out if the immersive VR mode is supported:

If the promise resolves, then you can add a button in the page to inform the user that they can enter into VR mode using the HMD they have. You can also query with an inline session to see if you can render something inline inside the page (also called a magic window).

ADAPTING YOUR RENDERING CODE TO ADD SUPPORT FOR THE WEBXR DEVICE API

If you have your Three.JS render loop set up for regular rendering on a 2D screen, you will need to adapt it to support rendering with WebXR. First, here is the basic flow of how rendering works with the WebXR Device API, whether it is an immersive or an inline session.

Let me expand on some of the boxes in this diagram:

REQUESTING A SESSION

The mode parameter can either be immersive-vr, immersive-ar, or inline. Remember to gracefully handle rejection in case the XR device is not available anymore in between the moment you queried the support and the moment you requested the session.

REQUESTING A REFERENCE SPACE

When requestSession promise is resolved with success, you can request a reference space using:

The various possible types and sub-types were discussed earlier in this article. If you want to specify a subtype, use this code:

I recommend that you request the least capable reference space needed for your experience, because it allows you to support a wider range of existing XR devices.

SETTING UP THE XR LAYER

I’ll not dive into details here because it’s well covered in the WebXR Device API explainer located here. However, I’m going to cover the specific example of having an inline rendering going on, and then the user wants to “upgrade” their experience to switch to the more immersed mode.

Right now if you want to have an inline mode and then switch to an immersive mode, you need two canvases: one for rendering your immersive content and one for your inline content. Each canvas has its own webgl context. The primary reason for two context requirement is that when you ask the GL context to be compatible with XR (using makeXRCompatible), the underlying implementation makes sure the context is created on the GPU where the HMD is connected. This is very relevant to computers with multiple GPUs, where the HMD may be connected to an external case containing an additional GPU.

Note: There is a discussion inside the WebXR Device API community to avoid the need of 2 sessions potentially, making it simpler to switch from one experience to another. Here is the issue where it’s being discussed. There is also work in progress to avoid having to request 2 sessions (one inline, one immersive) which would simplify quite a bit of the code.

Here is the code to set up the XR Layer with Three.JS:

ADAPTING THE RENDERING TO SUPPORT IMMERSIVE-VR RENDERING

Whenever you render with the WebXR Device API, the render loop needs to be updated. First and foremost, you’re not going to request a new animation frame on the window but on the session itself. The reason for that is because the XRSession will take care of calling your code at the right frequency depending on the recommended refresh rate for the HMD. In VR, the refresh rate is very likely to be different from the refresh rate of the main screen. For example, some All-In-One HMDs have a refresh rate of 72 Hz, while high end VR HMDs such as Oculus* Rift or HTC* Vive have a refresh rate of 90 Hz, and we expect to see HMDs with a 120 Hz refresh rate in the near future. One of the main reason VR experiences are rendered at higher refresh rate is to provide a smooth and more comfortable experience to the user (which also helps fight motion sickness).

Typically you will have something like this:

Here is how the render function would look:

The xrFrame parameter is the object carrying the information needed to render the current frame for the current XR device. Before you go ahead and render, there is a bit of bookkeeping to do with Three.JS to make sure the rendering works with WebXR:

The next step is to finally get into WebXR specific rendering bits. First, you need to bind the layers, which means you’re going to tell the Three.JS renderer to render into the XRSession layer:

Then you need to get the pose of the device (the pose is the rotation and position, if available):

The pose will give you all the information you need to render the scene for a XR device. Then you will need to render the views. The WebXR Device API has the concept of views which typically maps to the number of screens inside the VR HMDs. Typically, you’ll have 2 views (one for each eye):

From there you can iterate over the views and render each eye:

Rendering each eye will look like this:

The best part of the “views” approach is that you can keep reusing the same rendering code for your inline session, because the only difference (beside the matrices values) is that you’ll have only one view to render.

PUTTING IT ALL TOGETHER

This article has described how to render immersive web experiences for different types of devices. The next article in the series explains how to add support for VR inputs to make your experience interactive.

ADDITIONAL READING