How to Start Building Your Own WebGL-Based VR App
Use My Enhanced Boilerplate Code to Take Your First Easy But Powerful Steps
You’ve got a smartphone-to-VR headset like Cardboard or one of its many plasticized variants (that’s me — I’ve got a $30 VR Box). Or maybe you’re lucky enough to already have your hands on a Daydream. Or maybe you’re even luckier and have an higher-end headset like Oculus, Vive or Playstation.
Whatever your headset, you’re tired of using it to play with other people’s VR apps, and you’ve decided to write your own. This tutorial shows you how to get started in three big steps (with some little ones in between).
So, you’ve got two possible paths before you:
- Use the popular commercial solution, the Unity game engine, to create your interactive 3D environment.
- Use WebGL and the open source library, Three.js, to do the same.
Just in case you’re wondering what the hell Three.js is, it’s a library that simplifies the handling of JavaScript’s, WebGL API. And in case you’re wondering what the the hell WebGL is, it’s a native-JavaScript library that makes it possible to create 3D content in any compatible web browser (and most are), without the use of any extra plugins, tapping, where possible, the host’s GPU to do the renderer’s heavy lifting. It’s all sort of like putting a stripped-down, programatic version of Blender inside a web browser. Check out the Three.js example’s page to see what it plus WebGL can do, which is quite a lot.
There are numerous great debates online (example here) about which of these paths — Unity vs. Three.js — is the better one, but for this tutorial we’ll be taking the latter path. The idea is to get you coding quickly, with no signups of any kind, and you can weigh the Unity/Three.js pros and cons as you become more expert.
If you’re like me, and it’s been a while since you’ve dabbled in 3D for the web, I think you’ll be pleasantly surprised by how far things have come.
We’re going to take an approach that should work on any VR headset that’s capable of viewing a webpage, and we’re going to write controls that are highly generic (i.e, gaze for 3 seconds to select objects) so we don’t have to worry about the differences in physical controls between headsets, or the complete lack thereof (mine has none).
What You’ll Need to Complete this Tutorial
- A smartphone-to-VR headset. (Higher-end headsets like Vive, Oculus, Playstation, etc. will probably work too, so long as you can get them to view a standard web page, but I don’t include specific instructions for doing so as I don’t have any of these fancy headsets — yet.)
- A mobile phone, preferably with an onboard gyroscope. (Even one without might work, as some handsets “synthesize” a gyroscope’s output from accelerometer data, but results are likely to be shaky.) I’m testing on Android, both with and without a gyroscope, but an iPhone should work too.)
- The Chrome browser installed on your development computer and on your smartphone. (Other browsers will likely work too, but Chrome is what I’ve tested the code in.)
- Something in which to edit JavaScript code. (Atom is a good, free choice if you don’t have another code editor installed already.)
- A novice to intermediate knowledge of JavaScript. (Come on! Try it!)
Google’s WebGL VR Playground
Your first stop on the open source VR trail should be Chrome Experiments for Virtual Reality. There Google has put up some simple but inspiring sample apps. Give these a play, and then notice that they provide you with some boilerplate code to start building your own app (see “Developer Tools” at the bottom of the page).
Big Step 1: Download Google’s Cardboard starter code and give it a try on your headset.
To play with Google’s VR starter code, or your own app as it progresses, the easiest thing to do is run a local web server from inside the codebase’s document root directory (i.e., the directory containing the index.html file).
On OSX, this is very simple as nothing needs to be installed.
On Windows or Linux, you’ll likely need to install a web server app if you don’t have Python installed (see this article for help with that).
Once you have runnable local web server, you’ll do something like the following. These steps are OSX-specific, but should be easily adapted to the OS of your choice.
1. Unzip the download file.
2. Open Terminal and cd to www/cardboard/. For example:
cd Downloads/www/cardboard(If you then run ls, you’ll see the index.html file is listed — this is where you want to be.)
3. From inside that directory, run:
python -m SimpleHTTPServer 8000Now you’re locally serving up the Google VR boilerplate site.
4. To view that now-live site, open up the Chrome browser on your development computer and enter the following as the URL.
http://localhost:8000/5. If the index page doesn’t load completely, click View>Developer>Developer tools so you can see the error. It’s likely this:
(index):54 Uncaught TypeError: controls.rotateUp is not a function6. Comment out line 54 in your code editing application (we don’t need it anyway). Save and reload the webpage. You should then see something like the screen grab below in your Desktop Chrome browser.
Later, running a Desktop version of your in-progress app in this manner is going to useful for fast development. It can be made to, as the Google boilerplate does, respond to mouse clicks, so you can test most functions without having to don your headset.
7. Now you want to run the same webpage on your handset. For that, put your handset on the same Wi-Fi router as your development computer. Then find out your development computer’s IP address. On a Mac, for example, that’s done easily in the Terminal with:
/sbin/ifconfig en08. On your handset, open the Chrome browser and enter [your development computer’s IP address]:8000 as the URL. For example:
192.168.1.101:8000Up should pop the same web page as is running in your Desktop browser.
9. Have a play with that page in your headset. You’ll see that it’s a useful, but very bare-bones start. You’ve got a stereo render going on (two panes for two eyes). You’ve got a ground plane. You’ve got basic controls — the camera tracks your head’s rotation. And that’s about it.
To move forward with your app, whatever its purpose, it’s likely you’ll need to add at least the following key enhancements:
- Hands-free selections — gaze at something to “touch”/”hover on” it, and gaze at something longer to select it. This gives the user a simple but useful input mechanism that will work on any device. After all, without a means of input there’s no interactivity, just passive gazing, which gets boring pretty fast.
- Heads-up display (HUD) feedback — get info back (text, graphics, whatever) about the objects you’re “touching”/selecting. We need to be able to give users feedback about the objects they’re interacting with. There are different ways to do this, but we’ll use plain old HTML/CSS to add a 2D layer atop our 3D viewer. That way when a user touches something (with their eyeballs) we can give them info back about that something.
These pieces are pretty important in taking any app idea to completion. Let’s look at a demo first of these enhancements in action, then describe in detail how to implement them. Next stop: PluckyCharms Land.
Big Step 2: Try my enhanced starter code on your headset.
Pull your handset out of the your headset and punch in the following URL:
http://appbaan.com/vr_hud_and_selection_demo/index.htmlReinsert your handset into your headset, put it back on, and be transported to the magically delicious PluckyCharms Land, a trademark-friendly variant of the home of a popular breakfast cereal (only Fruity Pebbles rank higher on my list of favorites).
Charms (resembling very stiff mylar balloons) — an orange star, blue moon, pink heart, and green clover — float above a verdant, Irish-esque plane, while blue skies embrace us from above. Also, we can fly, but only in a very tight circle.
It’s a silly example, yes. (Palmer Luckey . . . Lucky Charms. Was that the subconscious association? I’m not sure. I didn’t think of that until I was writing this.) But the point is to use it to demo those key enhancements we discussed. As you’ll read below, the objects in the scene are all added via separate functions, so it’s easy to replace them with your own 3D building blocks and wire them up to the HUD and selection mechanism I’ve created for you.
So try out those enhancements. Note that if you glance at one of the charms, you get some info about it on the HUD. Just plain text in the demo, but you can draw anything you can make with CSS/HTML on your HUD — images, maps, diagrams, scores, etc.
The guide circle also turns into a progress bar upon first glance at an object and that object vibrates in response. You might instead alter the object’s color, make it glow, etc., as a signal to the user that the “touch” input has been received.
If you keep your focus on that object for 3 seconds, the progress bar will complete, counting as a selection. The screen will just flash green to signal the selection in the demo, but you could of course do anything you like with that callback in your own VR app — transition to another scene, or whatever.
If you glance away from the object before the progress bar completes, the selection count is restarted.
This is the standard sort of VR selection “grammar” that you’ll see in Samsung VR apps, Google’s demo apps, etc.
User input and app feedback. Basic stuff, yes, but the implementation of it isn’t necessarily obvious in these still early days of VR. So let’s now get to the nitty gritty of how to implement these features in your own apps.
Big Step 3: Download my enhanced starter code and walk through some of its key sections with me.
You can download the code straight out of the running demo on your Desktop web browser (by clicking View>View Source and then saving that page as a local .html file). Or you can download it from this Github repo. The latter option gives you all graphics, third-party JavaScript libraries, etc.
I won’t explicate the code here line-by-line — that would be stultifying boring — but I’ll talk about how all the big pieces work, so you can modify the code to serve your own ends.
The Google boilerplate is about 150 lines in total. My enhanced boilerplate is about 500 lines, but much of that is easy, but line-consuming CSS/HTML and standard Three.js-related functions to import the scene geometry and textures. The lines related to the actual selection-making and HUD drawing aren’t so many, so we can cover them in a bit of detail here.
Third-Party JavaScript Libraries Gratefully Leveraged
My enhanced starter code makes use of several third-party Javascript libraries, all available as GitHub repos (click the links below to access them):
- Mr. Doob’s Three.js (obviously) for the 3D scene building.
- Kimmo Brunfeldt’s progressbar.js for drawing the 2D circular progress bars that stylishly indicate selection states.
- Sole’s tween.js for the object animations triggered by “touching”. (You can indicate “touches” in some other way, in which case don’t strictly need this script, but it’s likely you’ll use animations elsewhere in your app anyway.)
Action Item: If you’re reading this post well after the publication date and your working from my GitHub repo of the demo code, it’s a good idea to refresh the source code for each of these libraries in your working directory. Download and replace all the .js files inside /js/third-party/. Note that there are several separate scripts for Three.js.
Building the HUD
There’s a load of CSS code that begins with the section below, in which we define a scene ID, which we’ll later apply to an HTML div element:
#scene {
position: absolute;
z-index: 1;
top: 0;
left: 0;
right: 0;
bottom: 0;
}The div assigned the above ID will hold the 3D scene we build with Three.js. On top of that scene we want to draw the HUD.
So below the scene ID definition, we define a series of other ids with z indices greater than scene’s 1. This ensures that the HTML div elements that get assigned these IDs will be drawn on top of our 3D scene.
In this manner, we build up a structure as diagramed below for each eye (left side only is shown, but the same structure is built for the right eye too).
Note that there’s one additional CSS/HTML combo not shown, just because it would make the diagram confusing, which overlaps the area of the #left_hud. This is used to layer a transparent green solid over everything (for a just a few frames) to indicate that a selection has been made.
We can select and manipulate these areas using standard JavaScript:
document.getElementById(“id_name”)The the div elements assigned the guide_circle IDs of course get the circular progress bars we’ll use to show selection state.
Inside the div elements assigned as text areas, we can put HTML paragraphs assigned the same CSS class. This lets us update both the left and right sides of the HUD at once using simple functions like this one for updating the bottom text:
function updateHUDTxt(msg){
x=document.getElementsByClassName("info_text"); // Find the elements
for(var i = 0; i < x.length; i++){
x[i].innerText=msg; // Change the content
}
}It’s about that simple really. Again, you can draw anything you like in your HUD. I’ve kept it simple in the starter code — just progress bars and text on the top and bottom.
Another approach is to make the HUD in Three.js as a separate scene. Jaakko covers that approach here (and says that it is required if you’re using Occulus, but I can’t confirm that).
Implementing Progress Bars to Show Selection State
Brunfeldt’s circular progress bars adapt nicely to showing selection states in a VR world.
One for each eye is initialized (inside the usual Three.js init() function in my implementation). That setup, mostly straight out of Brunfeldt’s doc, looks like this (with similar code not shown for the right eye):
left_bar = new ProgressBar.Circle('#guide_circle_left', {
strokeWidth: 10,
easing: 'easeInOut',
duration: SELECTION_TIME,
color: 'lime',
trailWidth: 2,
svgStyle: null
});The trailWidth parameter set to 2, paints a thin, white circle all the time, and makes for a good cross-hair-like guide.
Then we monitor for selections, which in a VR world take place when the camera is pointing right at an object (more about that below).
If the camera is not pointing at any object we’ve defined as selectable (more about that below too), we set the progress of the circular progress bars to 0 inside the standard Three.js render() function:
left_bar.set(0.0);//reset any active progress bars to 0
right_bar.set(0.0);This has the effect of resetting the progress bars’ animations to the beginning on every frame where no selection is taking place.
If the camera is pointing at an object we’ve said is selectable, we flag it as touched (explained below) and then in the usual Three.js animate() function, we start the progress bars’ animations:
if(left_bar.value() == 0){//don't restart progress bar if already progress
left_bar.animate(1.0, {
}, function() {
postSelectAction(object.userData.name);//add callback to left side progress bar to register completed selection
});
}
if(right_bar.value() == 0){//don't restart if in progress
right_bar.animate(1.0);
}Note that we only start the animations if the progress bars are at 0. That way we start up the progress animations only once — if the user keeps staring at an object, we want to continue the animations.
Note also that the left_bar gets a callback for the case that the animation completes. This gives us a mechanism to do something (whatever we want) if an object is stared at long enough for the progress bar animations to reach 360 degrees (that takes 3 seconds in the demo). We only need to call this function once, so we only add it to one eye, the left one in this case.
The 3D Scene
We strip out the endless ground plane added by the Googlers and we insert our own objects. With the exception of a few lighting tweaks, this is done with the drawSimpleSkybox() function and the drawShapes() function, which create primitives, load OBJ objects, and texture geometry in the standard Three.js ways. Replace or repurpose these functions to suit your own ends.
And thanks go to the creators of this tutorial for teaching me how to make a Three.js skybox in the first place and for providing the Irish-esque textures.
Implementing the Selection Mechanism
Above we described how to initialize and update the circular progress bars based on the scene’s selection state, but how to know when a scene’s object is selected or not? This section explains that. And it’s not as hard as you might think.
First we define the function below:
function getIntersections(objects){
var raycaster = new THREE.Raycaster();
var vector = new THREE.Vector3( 0, 0, - 1 );
vector.applyQuaternion( camera.quaternion );
raycaster.set( camera.position, vector );
return raycaster.intersectObjects( objects, true );
}Credit goes Falk Thiele who provided the innards of this function in a StackExchange answer about how to perform no-mouse selections.
What this function basically does (so far as I understand it) is create a test vector that shoots straight out of the camera lens. If that vector swings through any of the the objects in the list we feed the function, those objects are returned.
Empowered by this function, we can then then test for intersections inside the Three.js render() function. We call the function with a list of objects we want it to check for camera intersections (the charms OBJs in the case of the demo).
intersects = getIntersections(selectableObjs);Then if intersects.length equals 0, we know nothing selectable is being looked at, so we can set the progress bars to 0 as described above.
We can also set a flag on the object saying that no selection is in progress, which is useful for any special treatment we desire in the usual Three.js animation() function, like the vibration effect for “touched” objects shown in the demo. For flagging objects as touched or not, we’ll make use of a Three.js feature that lets you add and update custom user data to an object.
The case then for nothing selected inside the render loop looks like this:
if (intersects.length == 0){//Nothing being "touched"
left_bar.set(0.0);//Reset any active progress bars to 0 as nothing is selected
right_bar.set(0.0); //Loop over all OBJ objects (the charms)
scene.traverse (function (object)
{
if (object instanceof THREE.Group){
if (intersects.length == 0){//Set all charms' touch flags to false as nothing is selected.
object.userData.touched = false;
}
}
});
}
Then if intersects.length is greater than 0, we do pretty much the opposite.
else {//Something being touched
//Set the touched charm's touch flag to true, so we can give it special treatment in the animation function.
intersects[0].object.parent.userData.touched = true;
msg = getTouchMsg(intersects[0].object.parent.userData.name); //Update HUD text to register the touch
updateHUDTxt(msg);
}We set the touched flag to true and this will trigger special treatment inside animate(): The progress bars’ animations will be triggered and the vibration effect started on the touched object. With some repetition from above, that all looks like this:
if (object.userData.touched){
//Apply animation values from Tween effect (vibrate) object.scale.x = animScale.x;
object.scale.y = animScale.y;
object.scale.z = animScale.z; if(left_bar.value() == 0){//Don’t restart progress bar if already progress
left_bar.animate(1.0, {
}, function() {
postSelectAction(object.userData.name);//Add callback to left side progress bar to register selection.
});
}
if(right_bar.value() == 0){
right_bar.animate(1.0);
}
}
Going Fullscreen
Last niggle (promise): We need to make the Chrome UI elements go away in order to nicely view the scene inside our VR headset. The Googlers provide a handy function called fullscreen() for doing just that in their boilerplate. It’s just a matter of calling it somehow.
The code below from my enhanced boilerplate selects a small button (added in the CSS/HTML setup code) and puts a listener on it to call the fullscreen() function when it gets pressed.
var goFS = document.getElementById(“goFS”);
document.getElementById(“goFS”).style.display = ‘block’;
var doc = window.document;
var docEl = doc.documentElement;
goFS.addEventListener(“click”, function() {
fullscreen(docEl);
}, false);Then you’ve just got to make the button disappear when the browser is in fullscreen mode, and reappear when it’s not. There are probably more elegant approaches, but this code, again included in the enhanced boilerplate, does just that:
document.addEventListener(‘webkitfullscreenchange’, function(e) {
buttonState = document.getElementById(“goFS”).style.display
if (buttonState == ‘block’ || buttonState == ‘undefined’) {
document.getElementById(“goFS”).style.display = ‘none’;
}
if (buttonState == ‘none’) {
document.getElementById(“goFS”).style.display = ‘block’;
}
});Now Go Forth and Create Your Own VR App
OK, that was a 3,500+ word doozy, but with knowledge now of how to implement a HUD and selection mechanism you’ve got some basic tools to go forth and make your own VR app — say a stereographic gallery featuring Victorian-era content (selection mechanism allows users to choose their gallery of choice, while the HUD gives them feedback about the historical photos being viewed), or a spherical photo gallery (without all the nasty compression of Facebook), or an immersive data visualization (my next project).
Digging Deeper Tip: You can also dig into the code that the Googlers put up for their sample apps. You’ll see a lot useful nuggets in there, but there’s a lot going on, which is why I decided to start building my own basic pieces from scratch.
The possibilities are endless; and with more and more VR headsets being donned every day, your potential viewership, vast.
