Building a VR user interface for A-Frame City Builder
As early as I can remember I have loved playing with toy cars and trucks. Whether Hot Wheels, LEGO, or Micro Machines, if it had wheels I could use it for my elaborate imaginary city scenes.
This idea of a toy city is what inspired me to create A-Frame City Builder, a basic “city builder” style WebVR app as a side project starting late 2016.
Using the WebVR framework A-Frame I was able to quickly combine the whimsical artistry of Mike Judge’s city voxel models into an early proof of concept app that let the user place voxel cars and trucks using the HTC Vive controller in VR.
Making the “MVP” of City Builder
The code was not overly complex at first: the app simply consisted of a child entity on the controller which spawned a clone when the trigger was pressed. I quickly added a simple object switching feature which cycled to the next object from an array of models each time the trackpad was pressed.
At this point, I hadn’t thought about a user interface so the controls were minimal and utilitarian: the current place-able object was just “stuck” to the Vive controller model. Pressing the trackpad instantly switched to the next object in the array with no animation or fanfare.
I quickly found it felt more natural to have the object’s horizontal plane lined up to be parallel with the top of the Vive controller model.
It was nice to be able to see the object close to where my hands were holding the controller, like a master jeweler peering at a precise placement of a gemstone — now with voxels!
At this point the app was plenty usable. I was amazed with what scenes could be created given these two simple tools: clone and switch object.
I had only imported a few dozen models from Mike Judge’s collection, but already it was plenty sufficient to allow my friends and coworkers who tried out the app to construct fun scenes and tell unique stories with a few movements and presses of a VR controller. (Repo snapshot of this point.)
Finding a Better Solution to Scale
It was clear that using one button to iterate blindly through an array of objects would not scale well. Mike’s collection has over 400 models and I’d love to eventually allow people to upload their own voxel creations. So, I began brainstorming a more effective interface starting with some my own expectations:
As a kid building a VR city with voxel objects…
- I want to focus on my primary object that I’m placing. I want that front and center.
- I want to always be able to switch objects easily, but that’s not as important as placing the currently selected object so don’t get in my way!
- I want to be able to switch objects very quickly without having to move both hands. I want an interface that can use one-hand movement and not require a 2-hand gesture.
- I want to see many objects quickly. I want an interface that shows multiple objects at a time.
- I don’t want to “get stuck” at the end of a list. I want an interface that loops at the end of a list.
- I want to know something about each object, at least a name.
- I’d like to be able to figure out what button maps to the UI without having to read instructions.
I started sketching out some ideas on paper:
I tried using Tilt Brush and found it helpful for experimenting with 3D placement and sizing, but only after having started a creative direction. My initial ideas were most easily brainstormed and iterated with paper and pencil first.
However, after having a basic ideas on paper I found the A-Frame inspector was an excellent tool for rapid prototyping and iteration. I would very often tab back and forth between my prototype project, A-Frame inspector mode, and my codebase. A-Frame inspector was a tremendous time saver to make quick tweaks to positioning and rotation without needing additional round-trips of code changes and browser reloading.
I next tried building out a metaphor of a “watch” which is out of the way but still accessible with a twist of the hand — even going so far as to build a quick watch voxel model. (I had an idealistic image of a futuristic sci-fi hand like a portal gun mixed with Buzz Lightyear’s arm controls.)
The watch concept didn’t work very well. I found myself wanting to be able to see both the currently selected object and the menu at the same time. Moving my head back and forth to see the watch and the object was not a great experience.
So then I went simpler and literally placed object preview images directly adjacent to the controller and the currently selected object:
Once I threw out the need to match a real life metaphor, like a watch, it became much easier to directly address the design challenges.
I started refining the design to add some borders to the preview images and added styling to clearly indicate a “selected” state. The controls were really coming together!
One of my user goals was to be able to quickly and easily be able to see upcoming objects, and if I was really in a hurry to make a cool city scene it’d be nice to see ahead.
So I added 2 next and 2 previous object previews sandwiched around the currently selected object.
A 50% opaque background frame provided nice separation from other elements in my field of view, and at certain angles it has a nice specular sheen.
If I wanted to return to a skeuomorphic design concept I suppose at this point I could have experimented with designs that resembled a tablet (old style or the iPad variety), but I didn’t see any value in trying to cram that in.
In order to support hundreds and eventually thousands of objects, I wanted to introduce the concept of grouping objects together. You could browse objects of the same group by going forward and back within a group, or switch groups by going up and down.
I added some arrows to experiment with this concept:
I really liked this — seeing the arrows gave me some additional inspiration, wouldn’t it be great to be able to see where your trackpad is hovering before selecting an option? It turns out that the Gamepad API provides the raw axis position which can be interpreted into hover position, allowing me to put that together into a nice hover state:
Another fun thing about this design is that it can lend itself to at least 2 different methods of use to accomplish the same thing — one-handed quick via touch pad, or a “laser pointer” style like that used in Tilt Brush and other palette UI concepts.
I continued to add polish including quick animation when switching objects within the same group. I added labels for each object and the group name using the A-Frame bmfont text component (now part of A-Frame core).
Technical Challenges of VR User Interfaces
I have kept this article focused on the design decisions and steered clear of the technical side of implementation, but I must point out that by this point the vast majority of the development complexity of A-Frame City Builder was in fact building the user interface, not building of the logic of the game itself. An entirely separate blog post could be dedicated to the process of separating this select menu UI out of the base game code and into a standalone A-Frame component.
Another fun topic worthy of a separate thread was the process for making this component compatible with both the HTC Vive and the Oculus Touch controllers. Componentization was a key catalyst for being able to support both control schemes as it forced separation between gamepad events interpretation, user interface state management, and core game logic.
Once I had finished the separation of components, I was stunned to see the size of the select menu UI component— 40kb vs 11kb for the City Builder game logic!
As a result it’s absolutely clear that in order to enable developers to create the content we imagine necessary for VR to be successful, we need many more of these basic UI components available out-of-the-box.
The Finished Product
I have shared a repo for theselect-bar component including a bit of documentation and an example project. The API is inspired by the HTML5 select element: https://github.com/kfarr/aframe-select-bar-component
Here is the finished product in shiny YouTube glory:
You can take a look at the current A-Frame City Builder repo, including a link to a live online demo if you have a WebVR compatible browser with the HTC Vive or Oculus Rift: https://github.com/kfarr/aframe-city-builder
Closing Thoughts: Design Lessons Learned
Most VR applications seem to have a PRIMARY action and SECONDARY action.
Primary Action:
- What is your primary action? It usually relates to the core value statement of your application.
- Ideally a good primary action has minimal if any traditional “user interface”.
- The UI and is intuitive and/or explained clearly and immediately in interface.
- In VR, often the Primary action can be literal: painting, bow and arrow, shooting gun, placing object
Secondary Action:
- Often a secondary action is a modification of the primary action (change the brush, reload arrow, switch weapon, switch object) I find secondary action is actually more challenging to design than primary action! The secondary action design goals are tough to mix together:
- It must feel natural and appropriately related to the primary action
- It must be out of the way of the primary action
- BUT it also must be immediately available at a moment’s notice
- It likely involves choosing something from an array, or even an array of arrays
Putting it All Together:
Your user interface may be the most complex part of your VR app, so embrace it!
To close this post out, here’s an additional gameplay video of A-Frame City Builder. Please excuse the poor audio (using Vive mic) and poor frame rate (was having trouble with Firefox Nightly and WebVR and Chromium was out of commission).
