WebAR with WebXR API, Part 2
In this article, I would like to introduce some key pieces of code for the actual usage of the WebXR Device API at this point. I don’t want to explain all the source code of the actual 3D model viewer development using the WebXR Device API, so please refer to the full source code in View3D or WebXR Samples from immersive-web Github repo.
Check my previous article: https://medium.com/p/e191a2dc7177
Availability check
WebXR
The WebXR Device API has isSessionSupported, which an availability query method based on the type of XR session that you want to use. Therefore, it is possible to verify the availability of WebXR by first checking the existence of the WebXR Device API and then performing the isSessionSupported method.
If you want to use a specific feature in XR, you can verify the availability of that feature by checking the existence of the class that implements the feature in the window object.
SceneViewer / AR Quick Look
The fallbacks that I introduced in the previous theory section, can also be partially checked for availability before entering the session.
First, SceneViewer, an AR fallback application available on Android, is basically supported on all Android devices, as described in the previous article. However, the behavior is slightly different in the “ar_only” mode because the ARCore package must be used. Let’s say that you have a device that not supports the ARCore. If not in “ar_only” mode, it will be directly connected to the SceneViewer’s 3D model viewer. On the other hand, if set to “ar_only” mode, it will be directed to the ARCore Installation page, and you’ll see the message that just ARCore cannot be installed. This behavior makes it difficult for SceneViewer to determine whether AR is available until the user enters the session. Instead, ARCore supports Android version 7 and later and currently supports only the devices listed on the ARCore Supported Devices page, so if necessary, you can use the navigator.userAgent to check whether it’s Android, its version, and its name to determine if it is available.
For AR Quick Look, which is a fallback for iOS, you can check more simply using relList.supports(“ar”) method from HTMLAnchorElement. However, some browsers, such as Firefox, cannot use AR Quick Look even though the method returns true, so the browser should be limited to Safari and Chrome, which currently supports AR Quick Look in iOS.
Entering the Session
WebXR
After checking the availability of WebXR, AR sessions can be entered through the WebXR Device API as follows: As you enter an AR session, you can pass the functions you want to use through additional parameters, and you can also pass additional options such as specifying root elements for “dom-overlay” feature.
It is not written in this code, but additional code is required after entering the session. If you are using a 3D framework such as three.js, you must use WebXR setup methods likerenderer.xr.setReferenceSpaceType, renderer.xr.setSession, and if you do not use a 3D framework, you must create a rendering loop directly based on an XRSession and perform additional tasks such as setting up an XRWebGLLayer. See the following source code for an example.
SceneViewer / AR Quick Look
For SceneViewer, the session can be entered via an intent link that uses package “arvr.google.com”. Refer to the following documents for options that can be set up:
In the case of AR Quick Look, you can easily use it by linking the 3D model’s source URL (usdz/reality) to the anchor element href. You can either show the anchor element directly on the web page, or perform a click() method after generating the element like the following gist. It is important to note that there must be one <img> thumbnail element inside the anchor element. Otherwise, instead of entering the AR mode directly, it will enter Object mode, which is a 3D model viewer mode.
Check the following document for AR Quick Look options.
For SceneViewer and AR Quick Look, after entering a session, the rest of the AR-related actions are performed in each application, so you don’t have to pay more attention. Conversely, WebXR requires developers to do more jobs, such as controls and UI. This part can be customized differently depending on the situation, so let’s take a quick look at each example.
Initializing
After entering an XR session, the requestAnimationFrame method exists in the XRSession instance returned from the requestSession. The animation loop can be divided before and after the 3D model is placed. Prior to the placement of the model, the WebXR Device API’s hit-test function is performed based on the center of the current screen to prioritize the search for the floor surface, and once the floor is found, the 3D model can be displayed on the floor. After the model is placed, the main task is to update the controls in the viewer by frame.
By calling the requestHitTestSource method from the “hit-test” feature, you can obtain a hit test source to find the point of collision with the detected plane based on the center of the screen. When a hit test source is requested in the “viewer” space without any options, hit-test is performed in the direction same to the device’s camera from the center of the screen as the origin. This can be changed, see the hit-test explainer from the immersive-web Github for more information.
After obtaining a hit-test source instance, you can use it to perform a hit-test for each animation frame from the XRSession. The results like the location, direction, etc. of the corresponding plane can be determined through matrix inside XRPose’s transform property.
When you want to filter the plane pointing in a particular direction, you can use the direction of the coordinate system being converted by the matrix. For example, in the case of the matrix’s fifth element, it’s the same value to the new y-axis direction of the base coordinate system’s y-axis converted by matrix, so if this value is large enough, it can be determined that it is pointing upwards.
The following is an example of implementing the function using three.js’s Math-related classes.
Subsequently, the 3D model can be displayed on top of the collision point. Since a value of 1 in the WebXR coordinate system corresponds to 1m of actual space, it is recommended to calculate the size of the bounding box in the 3D model and adjust the scale of the 3D model accordingly.
AR Controls
In the general AR 3D model viewer, the parts that can control the model are largely divided into three categories: Rotation, Translation, and Scale, which correspond to the rotation, movement, and sizing of the model. In the case of rotation and scale, each action can be performed to the user’s input, as is almost the same as the typical implementation of the 3D model viewer’s control without AR. You can use the inputSources property within the XRSession for this, which will be added for each finger input.
For translation, further implementation is required. Because the movable area should be limited to the detected floor area, in this case, the point of collision with the floor surface should be determined using a hit-test again. For the first model display, you can always perform a hit-test on the center of the screen regardless of user input, but for translation controls, a different approach is needed because the model must be displayed at the point pointed by the finger.
You can use XRTransientInputHitTestSource for this. If you set its profile to “generic-touchscreen,” it can bring hit-test results at the point that the user is currently touching.
In addition, XRSession class have select event that occurs once at the start of touch, and selectend event that occurs once at the end of touch.
UI
If you use the WebXR Device API, you should customize the UI yourself. If you wish to use floor recognition-based AR, you can use UI elements under the 3D model that indicate the location corresponding to the current floor surface and the direction in which the 3D model points.
When displaying a 3D model on the wall or in the air, you can use the UI elements that indicate the location of the wall currently displaying the model, and a movable direction. It can give a more realistic impression of environmental blending.
Conclusion
Since the WebXR Device API is the quite latest API, there are many restrictions on the use of AR sessions and there are not many organized documents at this point. Hence many people might hesitate to create AR applications with WebXR Device API. However, if the number of browsers supporting WebXR API increases and API is finally established in the future, I hope to see many examples of WebXR Device API usage on the web, taking advantage of the ease of using powerful functions such as dom-overlay without the hassle of installing apps.
I hope this article will help anyone who finds a way to use the WebXR Device API.
