Chess Game using WebXR Device API

14 min readApr 19, 2020

Demo video: AR Chess game using WebXR Device API

Recap from previous article

My previous article Basics of Augment Reality, was to make you aware of various moving pieces in AR world, now you know few things about AR like What is AR?, Types of AR, high level of How AR works and finally the Good and the Limitations of AR.

AR gained huge popularity in last couple of years and all thanks to our mobile phones. AR requires some complex sensors, a display unit, camera and processing power. Our phones have all the awesome equipment already attached to it. This makes it easy for AR to reach masses, which is the cool mantra for success of any product. Some experts like Digi-Capital sees the great potential in AR. Due to AR’s easy accessibility, AR find it’s implementation in every aspect of our life.

Since, AR tech is already in your pocket all that’s left for the big players is to make use of it. First the big players are investing hugely on building SDK/platform/API to build AR apps and then there are product companies/individual contributors who are using these tools to make AR applications.

Popular tools for developing AR apps

Some other interesting tools:

I started exploring the AR world recently and found it very exciting. In my experience till now, developing an AR application is easy but choosing the right tool to build is difficult. And on top of it due to AR’s gaining popularity new AR development tools are coming in the market frequently.

In this article, I will discuss building an AR Chess game using one such tool WebXR Device API, here’s the video link for Chess game. In the end I will discuss why I am betting on WebXR device API for building AR apps.

Let’s get started.

Augmented Reality weds Javascript

Javascript is the most popular technology in 2019. It is used heavily in web application development and now it got one more magic in its sleeves WebXR Device API.

I was thrilled with the idea of marrying a website with AR/VR experience. It opens up the pandora box for ideas of marvelous user experience possible from this collaboration.

WebXR Device API

WebXR represents a set of standards that defines the rendering of 3D objects into a screen, either to show the virtual world (VR) or to bring 3D objects in the real world (AR). The WebXR Device API provides the functionality which is necessary to enable developers to build immersive web applications across wide variety of hardware.

WebXR uses one of the stable graphics API WebGL, which is used to render interactive 2D/3D models in a compatible web browser and is present in the market since 2011. As an AR developer we might not be using WebGL directly and will use JS libraries to render 3D objects. There are lots of libraries out there but Three.js is the one widely used. The reason for using an external library is, WebGL does not provide virtual camera view and libraries like Three.js implement virtual camera functionalities used to render scenes in AR/VR.

2. Standardize WebXR by Mozilla, Google, Microsoft and others

Mozilla was working on bringing Mixed Reality to Web and proposed the standard for WebXR Device API. Top developers from Google, Microsoft, and other big companies join Mozilla to create WebXR Device API and giving WebXR support to their respective browsers. Apple had not yet announced his stand on giving WebXR support in Safari.

3. Current State of WebXR in Chrome, Firefox and Edge

WebXR is supported by Firefox Reality, Chrome Desktop and mobile > v79 and Edge Desktop > v79. Here you can track the latest browser support for WebXR Device API.

Bear in mind browser support is partial support which might change in the near future, with development in WebXR API. WebXR tech is currently in the development phase and Chrome/Edge/Firefox is working on implementing new WebXR features in their browser releases.

Enough of theory, let’s get technical

For getting my hands dirty on WebXR, I used the chrome browser in android phone.

Note: Since Web XR is an emerging technology, all the tech concepts discussed from now on are subjected to change. I will try my level best to keep this section up to date with the changes done in WebXR Device API

We will be building an interactive Augmented Reality Chess game using WebXR. I had attached the necessary links in the Resources section, at the end of this article.

1. Building an augmented reality (AR) application using the WebXR Device API

This article goes hand in hand with Google’s wonderful explanation for Building an augmented reality (AR) application using the WebXR Device API. Go through the article and try to set up your machine for AR development using WebXR. I am using Chrome version 96.0.4664.104 in android and WebXR works for me like a charm.

Once you have gone through the above article I hope your development set up is done and you are aware of few WebXR concepts. A quick recap of those concepts:

navigator.xr && navigator.xr.isSessionSupported && await navigator.xr.isSessionSupported(“immersive-ar”)

To verify whether your device supports AR and necessary XR features we need to look for navigator.xr. Another check requires to verify the support for immersive-ar mode support. If the device is not supported, unsupported browser message is displayed.

Requesting session on device

On the DOM click event we will request for XRSession in immersice-ar mode. A canvas context is created using webgl, making it compatible to render 2D/3D models in HTML canvas element

Session start

When the session started we will have an instance of XRSession. Session baseLayer will also be updated using XRWebGLLayer, this is an interface between WebXR device and WebGL context. We will also start render loop for rendering objects at 60fps via requestAnimationFrame using Session.

Now it’s time to put Three.js in use and initialize renderer, scene and camera. Other things can also be added here like creating reticle and for our Chess game initializing chess pieces and loading all the Chess pieces models.

Each frame refresh

Here requestAnimationFrame will be called again with each frame refresh function, which will start the render loop. This is the place where you keep the code that should run in each frame refresh. Eg, Tween can be updated here, which will make the animation smooth while interacting with 3D models.

That’s it simple right. Not so, for the first time, but trust me it will be a cakewalk once you get the hang of it.

AR has lots of concepts to go through like understanding real world coordinate systems, playing with camera and scene, rendering 3D models, updating their pose, projection, interacting with models, adding light and shadow to the scene, etc. I tried below to help you kickstart with a few of these concepts. For these concepts I will provide link to more detailed information page.

2. AR coordinate system

Understanding AR coordinate system is super important. In AR camera will always be fixed in a position and the world moves around the camera. Three.js has PerspectiveCamera which uses perspective projection to mimic a human eye. The camera is not the mobile device camera, it is the virtual camera which will show the rendered 3D model. Each model needs to be at their respective position and all the positions are calculated based on the above coordinate system. In Three.js using axis object you can actually see the coordinates in the real world, which will help in positioning objects.

The AxesHelper will show color-coded lines (axes), x-axis is red, y-axis is green, z-axis is blue.

Axis names were added in photo for reference, they will not appear via AxesHelper.

Resources:

3. Playing with Matrices

Our good old matrix multiplication is used to transform, skew, translate, rotate and scale a 3D model. In 3D graphics a 4 X 4 matrix is used, this helps in transforming (x,y,z,w) vertices.

If w == 1, then the vector (x,y,z,1) is a position in space.
If w == 0, then the vector (x,y,z,0) is a direction.

There are lots of magic which can be done from the matrix, I will encourage you guys to go through Opengl tutorials. This tutorial explains the four coordinates Model Coordinates, World Coordinates, Camera Coordinates and Homogenous Coordinates and their relationship. Also, it explains how 3D world is projected in a 2D screen using Projection Matrix.

Resources:

4. AR Chess introduction

I am fascinated by the chess game, though I am not so good at it. After creating AR Chess I learned a lot about AR concepts, WebXR, rendering 3D world, raycaster, adding light and shadow in rendered scene, using external libraries like Three.js and Tween.js, etc. I thought of sharing my experience with everyone.

Firstly, as you might already have seen in the Chessboard image above, I am not using typical chess pieces just to make game little interesting, so I would like to thank the Google Poly project and to the awesome 3D designers who developed these models.

Evol for model Zombie
Franck Corroy for model Année 30 — Dame au manteau
Joe Tossey for model Warrior
Joe Scalise for model Storm Trooper 7 V1
Jarlan Perez for model Chess Rook
Marr Chal for model Minotour_Test001

Now, since we got the knowledge of how WebXR works, time to bring an important and awesome library Three.js.

5. Three.js

It’s a JS 3D library, aim to create an easy to use, lightweight, 3D library with a default WebGL renderer. The library also provides 2D Canvas, SVG and CSS3D renderers.

There is an awesome upcoming book on Three.js by Lewy Blue, Discover Three.js. It starts from very basic assuming you don’t know anything and in the end it will make you 3D world champion. A particular chapter YOUR FIRST three.js SCENE: HELLO, CUBE! will explain at a high level about all the moving pieces required to give life to a 3D model i.e. renderer, camera, scene, display, mesh, texture, etc.

To experience how powerful three.js is have a look into three.js example projects.

I am gonna discuss few features which I used in my Chess game:

5.a. Add 3D models

There are 2 ways to add a model in the scene; create the model using three.js geometry or load the model.

Create the model

Chessboard model is created in 3 simple steps:

1. Using BoxGeometry created the cuboid and divided into squares. Since the cuboid is hollow there are 12 faces. Every face of cuboid was divided into black and white squares (actually triangles, and 2 triangles with same color make a Chessboard square).

const cbgeometry = new THREE.BoxGeometry(BOARD_SIZE.width, BOARD_SIZE.height, BOARD_SIZE.depth, 8, 0, 8);

2. Time to add materials to the cuboid, MeshBasicMaterial is used to add black and white materials on each square and all the 12 faces. Three.js provides a way to add Textures also.

const cbmaterials = [];
cbmaterials.push(new THREE.MeshBasicMaterial({
color: 0xffffff,
side: THREE.FrontSide,
vertexColors: true,
transparent: true,
opacity: 0.8
}));

3. Create a Mesh using cbgeometry and the material. It’s a good idea to wrap the mesh into a Group before adding to the scene. Our empty chessboard is ready to go into the scene, before rendering the board in the scene we will add all the chess pieces (which is discussed in section 5.b. below)

const chessBoardMesh = new THREE.Mesh(cbgeometry, new THREE.MeshFaceMaterial(cbmaterials));const group = new THREE.Group();
group.add(chessBoardMesh);

Load the Model

Another way to add the model is to load them. To load all the awesome Poly models you can use OBJLoader and MTLLoader (used to load respective materials). On downloading Poly model there is an option to get the OBJ file. This will contain a “.obj” (object file) and “.mtl” (Material) file.

From Google’s codelabs link, which you have already gone through, you must have downloaded a project “ar-with-webxr-master”. In “shared/utils.js” you will find a function “loadModel” where the loading model functionality is already set up. This function will return a single promise which will load a single model. For Chess, we need to load 6 models, so I did a small tweak in function “loadModel” and created “loadAllModels” which will take all the model details and load them all at once using Promise.all.

5.b. Render 3D models

Rendering 3D models is to display 3D models in the real world. To do this, update the scene object and our render loop (which started running in “Session Start”) will take care of rendering the scene in the real world. Voila your 3D model has now seen the light of the day and will be displayed in the real world, coming from camera feed.

For the Chess, till now we have our Chessboard created and individual chess pieces model loaded. Now, it’s time to position the pieces into respective squares in Chessboard, using Vector3.

let pawnPosition = new THREE.Vector3(
BOARD_SIZE.width / 2 — boardSquareSize / 2 — i * boardSquareSize,
BOARD_SIZE.height / 2 + pawnSize.y / 2,
BOARD_SIZE.depth / 2–3 * boardSquareSize / 2
);this.pieces.black.pawn[i].position.set(pawnPosition.x, pawnPosition.y, pawnPosition.z);

Once, the position of chess piece was calculated, then add the model in the Chessboard model, created before.

board.children[0].add(this.pieces.black.pawn[i]);

Do the same for all the Chess model pieces, keep adding them into the board, by calculating their respective positions. Finally, the scene is updated with the board added to it. Our render loop will kick in and it renders the scene in mobile screen.

this.scene.add(board);

The line below will add the board in the reticle (anchor) position, we will learn about reticle in the next section.

board.position.setFromMatrixPosition(hitMatrix);

For finding the position of each chess piece it’s easier if we start from the origin of our coordinate system. To do that comment above line and bring back our Axes in the scene.

this.scene.add(new THREE.AxesHelper( 10 )); // This will show you the 3 axes in screen, used to position 3D models.

Now the board will render at the origin of our Axes (0,0,0). Here’s how it looks.

5.c. Reticle

Reticle is a cursor which always stays in the centre of the screen. In some mobile games, you might have seen a crosshair at the centre of the screen, that’s the reticle. In our AR Chess application reticle is used to identify surface using raycaster (we will cover in the next section). In the project, you downloaded previously, code for Reticle class is already present in “shared/utils”.

In the update function, use setFromCamera to cast a ray from the centre of the screen (0,0)

this.raycaster.setFromCamera({ x: 0, y: 0 }, this.camera);

Now time for requestHitTest on the XRSession created before. RequestHitTest returns a promise that resolves with an array of XRHitResult objects, if a ray cast into a scene intersects a surface, in this case a horizontal or vertical surface in the real world.

const hits = await this.session.requestHitTest(origin, direction, frameOfRef);

Once a surface is hit i.e. detected, the reticle will be visible on screen

In Chess game initially reticle will be white in color and once all the models are loaded, reticle color will change to green.

5.d. Raycaster

As the names suggest, it casts a ray from an origin position, in a particular direction. The origin can be a position of tap on the screen, or it can be customized position (like in case of reticle center of screen).

Raycaster can work hand in hand with requestHitTest. We need to tell raycaster from where to start the ray and in which direction, setFromCamera is a perfect tool to do that for us. We pass origin, direction and frame of reference to requestHitTest and it will cast the ray. We will get an object of type XRHitResult, which helps us know if we hit any object in 3D space.

this.raycaster = this.raycaster || new THREE.Raycaster();
this.raycaster.setFromCamera({ x, y }, this.camera);
const ray = this.raycaster.ray;
const origin = new Float32Array(ray.origin.toArray());
const direction = new Float32Array(ray.direction.toArray());
const hits = await this.session.requestHitTest(origin, direction, this.frameOfRef);if (hits.length) {
  const hit = hits[0];
  << do whatever you want with the hit object>>
}

Keep in mind the ray will not stop after hitting one object it will continue to travel in its path and will return all the objects it hits, in the order of their depth. You have to decide which object you want to capture.

Perform a click event using Raycaster

Raycaster is also helpful to capture object(s) when DOM events are triggered, example get the model which is clicked. Here’s a stackoverflow example, how click event can be performed on 3D object. The key function is intersectObjects which returns the intersects sorted by distance, closest first.

In Chess game it is used to either pick a chess piece or move it to a chess face which is one of the square boxes.

Here’s the code snippet:

document.addEventListener(‘mousedown’, this.onDocumentMouseDown, false);onDocumentMouseDown(event) {
event.preventDefault();
const tapPosition = new THREE.Vector2(
  (event.clientX / window.innerWidth) * 2–1,
  - (event.clientY / window.innerHeight) * 2 + 1
);
this.raycaster = this.raycaster || new THREE.Raycaster();
this.raycaster.setFromCamera(tapPosition, this.camera);
const intersects = this.raycaster.intersectObjects(this.scene.children, true);
const intersect = intersects.find(model => model.face);
if (intersect) {
  <<do stuffs here either select a piece or move the piece>>
}
};

5.e. Giving smooth animations using Tween.js

Tween.js is useful to give easy and smooth animations to 3D model objects. The library’s engine has only one job to perform i.e. tween properties. Here are the features you will find in tween.js like you can control tween(s), multiple easing functions and callbacks.

Code snippet from tween.js README:

const coords = { x: 0, y: 0 }; // Start at (0, 0)
const tween = new TWEEN.Tween(coords) // Create a new tween that modifies ‘coords’.
.to({ x: 300, y: 200 }, 1000) // Move to (300, 200) in 1 second.
.easing(TWEEN.Easing.Quadratic.Out) // Use an easing function to make the animation smooth.
.onUpdate(() => { // Called after tween.js updates ‘coords’.
// Move ‘box’ to the position described by ‘coords’ with a CSS
translation.
box.style.setProperty(‘transform’, `translate(${coords.x}px,
${coords.y}px)`);
})
.start(); // Start the tween immediately.

Just make sure you are calling TWEEN.update(); in your render loop.

If you have reached till here time to give pat in your back. That’s all I wanted to share with you all. Please go through the resources section below to download the AR Chess game, if you haven’t till now.

I had fun building the Chess game using WebXR Device API, hopefully, this article is of some help to any enthusiastic AR developer looking to get his hands dirty in WebXR.

WebXR Device API advantages

Awesome developers from Mozilla, Google and Microsoft are working on building WebXR Device API. Though it is still very young, but I feel it has huge potential to capture a good segment in the market.

AR experience can be easily integrated with your existing (or new) web application
Making it standard across the browser is great. Build once, use in multiple browsers/devices
No need to install any app. Only your phone hardware should support AR
Updating app is super easy
Use the power of Javascript to provide AR experience
It’s free to use

WebXR Device API limitations

Tech is still under development and many awesome features are yet to be built
Heavy models eat up bandwidth and take time to load. However, there are few options available like, efficiently design the application so you load only models which you need them or compress the model; load it and uncompress in application.
I was able to detect only horizontal surface as of the time of writing this article
Apple is still thinking on whether to support Safari with this API or not
Opens up the door to discuss about privacy and security of using AR in web applications, since the camera will scan the area, so in sensitive areas, it would be tricky to use.

Resources:

Chess demo: https://youtu.be/6MPfq2_Pt00

Github: https://github.com/Brijesh1005/webxr-chess-game

Heroku site host link: https://ar-chess-webxr.herokuapp.com/