Get started with Augmented Reality on the web using Three.js and WebXR | Part 2
From zero to hero with AR on the web.
This post is the second part of a project tutorial where we create an augmented reality application with WebXR and Three.js.
Need to catch up on part one? Click here to check it out.
Welcome back! 👋
In the previous part of this project, we looked at the basic workings of Three.js, got our development environment set up, and got as far as rendering a spinning red cube in AR using WebXR and Three.js.
While this is already pretty great on its own, we won’t stop just yet!
In this part, we will cover some more advanced material such as hit testing and rendering of pre-made models with Three.js. The outcome of this will hopefully feel like a more “complete” AR experience, and will also provide a good base for reuse in other projects down the line.
Ready? Open the project back up in your IDE and let’s get right to it. 🚀
Hit-testing, hit-testing, 1..2..3!
To add some interactivity to the application, we need to implement some basic hit-testing. This concept might be unfamiliar to people not acquainted with graphics programming, so let’s first look at what exactly hit-testing is:
“In computer graphics programming, hit-testing is the process of determining whether a user-controlled cursor (such as a mouse cursor or touch-point on a touch-screen interface) intersects a given shape, line, or curve drawn on the screen. This may be done for movements of the pointer or the underlying shape, or restricted to user-initiated selection, such as a mouse-click.“
— W3C Wiki https://www.w3.org/wiki/Hit_Testing
In the context of this project, the hit-testing will enable us to detect where in the AR space to place an object once the user touches the screen. This is by far the most complex part of the app, but we are going to step through the process bit-by-bit so that we can get a good understanding of what is happening behind the scenes.
It’s also worth mentioning that the hit-testing operation will be performed on every render, so I’ve separated the relevant code into a separate file, to avoid cluttering up the main render loop with too much logic.
Begin by opening the file src/utils/hitTest.ts in the project folder.
This should contain an empty function called handleXRHitTest.
Add the below code to the function, and let’s look at what is happening here.
NOTE: The below code is adapted from the official Three.js hit-test example for WebXR. Credit where credit is due! 👏
We begin by first getting the current reference space and session from the xrobject on the renderer, and then declaring a variable to keep the value of our hit pose.
The concept of reference spaces is an important (and somewhat complex) concept in WebXR. Luckily, we can get by with a surface understanding to make use of them here. In the context of this app, we can think of the reference space as the local coordinate system of the AR space. The device needs to know about its surrounding space, to translate between the coordinate space of the Three.js scene and the real world. This ensures that objects are rendered in the right spot in AR, relative to the position of the physical device.
The session variable is simply used to keep a handy reference to the current running XR session, which will let us ask for information like the position and orientation of the user’s device at any given moment.
We also keep the last hit test result in the variable xrHitPoseTransformed, which we update every time a new hit test result is ready.
In the next step, we start by checking if a session exists and if a hit-test source has already been requested. If this is the initial request for a hit-test source, we request a reference space of type “viewer” from the XR session. A viewer reference space is suited for inline experiences or simple AR viewer applications, which should be a good fit for what we are doing here.
If our reference space request is successful, we move on to requesting a hit test source from the session, passing in the reference space we got. If this is successful, we finally assign the hit test source to the variable hitTestSource, and toggle the hitTestSourceRequestedflag to true.
Now, on to performing the actual hit testing! 💥
In the next step, we verify that a hit test source has been set before proceeding. If we have a source, we get the hit test results from the current frame, which we pass in as a parameter to the function. The call to getHitTestResultsreturns an array, and so we check to see if there is anything inside by looking at the .length property. If this check evaluates to true, we get the actual hit from this array, which here is always at the first position.
The next step is another check to verify that the hit object is not null, and to ensure that our reference space is correctly set. The null checking throughout this function might seem a tad excessive to some, but it is actually quite necessary as several of the API-functions could suddenly return null if the app loses connection with any of the XR systems. The last thing we want is a crash, so this is a more sustainable alternative.
If everything checks out OK, we get the pose by calling getPose()on the hit object, passing in our reference space. This will return the position and orientation of the hit relative to the device. We then get the transform matrix, and finally, we return it back through the onHitTestResultReady function so we can use it to position objects in our scene.
Now admittedly, this whole process is a bit convoluted — but you can rest easy because we now have all we need to perform hit testing for the app.
As mentioned, we are going to call this function on every render. So you might already have guessed it — we are going to use the hitTestHelper-function inside the render loop. And now that we’ve gone through the difficult part of setting up hit-testing, I think we deserve to take a shortcut.
I’ve included a function in the project which will create a simple circular marker mesh that we can display on any surfaces discovered by hit testing. If you’ve ever used any other mobile AR-apps, this kind of user experience might feel familiar.
The function to create a marker should already be imported into the scene.ts file. Update your createScenefunction to include everything in the snippet below.
You’ll notice the code related to our spinning Box mesh from part 1 has been removed. This is intentional, as we won’t be needing it going forward.
With the above changes to our renderLoop -function we are now performing a hit test on every iteration in the render loop.
The handleXRHitTest-function has two callback parameters, one which is called when a hit test result is ready, and one that is called when the result is empty. With this available, we can now show or hide the circular marker on any surfaces detected by the XR system, depending on whether or not the latest hit test is successful.
If the hit test is successful, we set the visibleproperty on the marker mesh to true, and then we update the position of the marker to reflect the result of the hit test. This will make it appear as if the marker is tracking the ground in real-time.
On the other hand, if the hit test is empty we simply hide the marker to indicate to the user that no surfaces were found.
Give the app a spin again— you should now see a marker being rendered on any surfaces detected by the hit testing.
If nothing is showing up, try moving around the room a little while moving your device in a circular motion to scan the surfaces in front of it. Reloading the page and re-entering the AR view might also help.
Now that we have a way to identify surfaces in the AR space, it’s time to finally release the koalas!🐨
Placing a model in AR space
A benefit of using a dedicated library like Three.js is that common tasks like loading and displaying a pre-made model are as easy as pie.
Three.js comes with a bunch of different loaders for various model formats, such as OBJ, 3DS, and gLTF. We are going to use a model in the gLTF format for this app, as this format has a good “quality-to-filesize” ratio making it a reasonable choice for a mobile experience.
As mentioned previously, the included model file has already been imported into the scene.ts file and is ready to use in our code. I’ve also imported the gLTF loader in Three.js that we will use to load our model. Let’s start by doing just that.
Add the following lines inside the createScenefunction.
On the first line, we are declaring a variable to keep our koala model in.
Then we create a new instance of the GLTFLoader, which we then use to load our model. The first parameter in the load function is the URL to the location of the model. The second parameter is the onLoad-callback, a function to be called once the model is ready. Here we simply create an anonymous function that will allow us to get data from the loaded resource. Inside this function, we then finally assign the model to the variable declared earlier.
In this specific gLTF asset, the model we will be using is stored as the first child on scene.children . A gLTF file can contain a bunch of different data, such as animations, cameras, and scenes. It is therefore important to know how your asset is composed. If you are using a downloaded model in the gLTF format, you can use this handy sandbox app from BabylonJS to inspect the file and figure out how it is put together.
Adding a WebXR Controller
Before we can react to touch events performed on the AR view, we need to set up a WebXR Controller and attach a listener to the “select”-event, which will be called whenever a user taps the screen while in AR-mode.
Begin by adding the following lines inside the createScene function
Then, add a new event listener for the “select”-event on the controller you just created, and define the accompanying listener function like this:
Inside the onSelect-function we first check if the marker is currently being shown. If it is, we can assume that the latest hit-test result represents the position of a valid surface to place our model on. We then create a clone of the model we loaded earlier and assign it the position of the marker. We also rotate the model randomly on the y-axis, so that the models won’t all be facing the same direction if we place more than one. This is a simple but effective way to provide a bit of variation to the scene.
Finally, we ensure the model is visible before adding it to the scene.
You can try running the app now, but you will quickly notice that we are missing a crucial part. Since we have no light added in the scene, the koala is rendered with a pitch-black color!
That won’t do — we need a light! 💡
Luckily, this is an easy fix. You can add a simple AmbientLight from Three.js to the scene with the two following lines
An AmbientLight will provide a global and evenly distributed illumination to the scene. We pass in the color white as the first parameter and set the intensity to 1. This will make sure our models have a crisp and clear look when rendered, and our koalas should finally show up in their full glory.
Now if you run the app, you should be able to scan the environment and place cute little koala bears around the room. 🐨
It should hopefully look a little something like this:
Final words
If you made it this far — congratulations! 🎉
You have now been through the steps needed to create an AR application using the WebXR API with Three.js.
If this project piqued your interest in AR and WebXR, feel free to use this project as a base to create your own AR experiences. You could start by animating the koalas! Or make them interactive in some way! You could also easily swap the koala model for something entirely different, simply by loading a different model file. Would you rather render a dog? Or perhaps a piece of furniture to see how it looks in your living room? Go hog wild!
I also want to mention that this little app is only showing off a tiny part of what you can do with Three.js. The library is capable of creating powerful and sophisticated 3D experiences, and if you enjoyed this little foray into 3D I strongly recommend having a look at the Three.js docs to further explore the library. I’ll leave some links down below to some additional material.
If you want to check out the completed project code, it is available on the “complete” branch in the project repository:
https://github.com/sopra-steria-norge/get-started-with-ar-web/tree/complete
Thanks for reading! 👨💻👩💻
PS: The WebXR Device API is still in development and might change over time. Anything not working correctly? Drop me a comment!
Additional resources and further reading material:
The big list of Three.js tips and tricks
Google WebXR Experiments [video]
WebXR Device API browser support overview
Three.js-journey [paid course, but very worth mentioning!]
