From Idea to Demo: Your VR Development Roadmap

Vivid graffiti that springs to life. Dreams where the world shifts when you look away. An empty space station that straddles three different timelines. Virtual reality has the power to bring us into totally new worlds of the imagination, where the only rules are the ones you invent.

But these worlds don’t just spring out of nowhere. As many people break into VR development for the first time, they’re discovering tricks of the trade ranging from architecture to animation to interaction coding. And even veterans of the game industry are changing how they think about the spaces they build. Once just images on a screen, these are now places beyond space and time that people can actually inhabit — at least until it’s time for a bathroom break.

With the 3D Jam just around the corner, there’s never been a better time to sketch out your own VR development roadmap. In this post, we’ll look at how VR developers can bring their projects to life, from idea to fully realized demo. Along the way, we’ll also dissect the development process behind Martin Schubert’s Weightless, the second-place winner of last year’s 3D Jam.

Step 1. Starting with Strength

During the brainstorming phase, it helps to start by breaking down strengths and weaknesses. Unlike any other kind of experience, VR gives you the ability to easily look in any direction and judge depth and distance at a glance. It’s not so good at things like moving around in a real-world space or comfortably reading text with small to normal font sizes. (You wouldn’t want to read a book in VR!)

Similarly, Leap Motion is a relatively new technology with the power to bring your hands into VR and reinforce your sense of body presence. As a touchless interface, it doesn’t provide any haptic feedback, and it can only reliably track what lies within the device’s field of view. Here are some essential resources you should read when thinking about designing for the Leap Motion Controller’s strengths:

It also helps to think about your own strengths, either as an individual or a team. Like many developers on the edge of the VR revolution, Martin doesn’t come from an engineering background. He has a degree in architecture and worked as a copywriter for an ad agency, until VR resurged and he wanted to become a part of it. After teaching himself Blender, Unity, and Playmaker (more on that later), he built Weightless over the course of 6 intense weeks.

Every developer new to the VR scene brings their own set of skills to the table. For Martin, it’s architecture — the creation of spaces that we can inhabit and move around in. In starting his VR development, Martin identified two essential elements of the Leap Motion platform that he wanted to build upon: fingertip interactions and lack of haptic feedback.

Fingertip interactions are unique to hand tracking and allow fine-tuned physical interactions that are profoundly unlike any other input platform. And while the lack of haptic feedback can be a barrier (requiring visual and auditory feedback), it can also be freeing.

Martin’s solution? Weightless objects that transfer momentum.

Step 2. Resources, Assemble!

Before you begin prototyping, make sure you check out the resources that are currently available, especially when it comes to interface design. Our Core Assets pack for Unity provides some fundamental UI widgets that you can quickly drop into a scene as a proof of concept. With a little love, you’ll be able to weave these into your experience and save some major headaches in the process. Check out the example scenes in the assets and try them for yourself!

Next up, if you don’t eat code for breakfast, scripting can be a daunting challenge. For designers and artists looking to break into Unity and VR, we recommend Playmaker — an incredibly powerful visual scripting tool that features a huge library of prewritten scripts. You can build interactive levels, AI behaviors, and gameplay mechanics just by connecting digital strings in the right places.

Step 3. Building Your First Prototype

The first prototype you build will be quick, simple, and ugly. The original prototype for Weightless was a grey box with a few shapes and the gravity turned off. This allowed Martin to quickly and easily test the core concept of the demo — being able to touch floating objects.

On the other hand, the starting concept behind our spaceflight demo VR Cockpit was set design. Real-world cockpits are incredibly complex, and we wanted to provide the same wonderful sensation of flight with a more streamlined set of interfaces. We already had the buttons and sliders and text displays as assets from our Planetarium and Widgets projects, so we needed an environment where these could be taken to a whole new level. As a result, we prototyped spaces first, rather than interactions.

Rapidfire set prototypes for VR Cockpit. Our team designed these in MAYA and exported them to Unity. Drop in the Leap Motion Core Assets, and you can look around inside a virtual space in less than five minutes.

Your prototype doesn’t have to be beautiful. But is it compelling? If not, you may want to try a new combination of ideas. It’s OK to try multiple approaches and scrap what doesn’t work. This is the most exciting stage — the canvas is blank, you’re full of ideas, and anything is possible. Once you hit upon a compelling prototype, it’s time to ask what it can become.

(A quick disclaimer: this series will follow through the design process of Weightless, but don’t feel like you’re bound by the sequence of steps! Every VR experience is different, depending on what you want to achieve, so do what makes sense for you.)

Step 4. Create a Narrative to Drive Interaction

Now that you have a core concept, it’s time to give the user a story. Just ask yourself three questions:

  • What is the user doing?
  • Why are they doing it?
  • How will they know this?

In the case of Weightless, Martin started with the idea of weightless objects floating around in a space. He then took it a step further by adding the core interaction — sorting space debris into two categories.

Why? Because the user is a commander aboard a salvaging space station. How do they know this? Because the environment sets the scene, and the station’s AI guides them.

Step 5. Build a Simple Interaction Loop

In gameplay as in music, repetition is the magic elixir that gets people into a groove.

Most successful games work by balancing challenging game mechanics with repetitive behaviors to get users into a flow. This is the bulk of what the user will actually do within the demo.

Designing an interaction loop can be a real challenge, and require several different iterations to get the right recipe. Each individual interaction needs to be compelling and intuitive in its own right, and be distinct from the other actions in the loop. Be sure to keep it simple at first — you can always expand on the loop organically as the user progresses through the game, allowing them to create more intricate sequences and access extended gameplay.

In the case of Weightless, the loop is:

  • Press the FILL button to bring space debris into the cabin
  • Inspect objects and determine whether they’re human or nonhuman
  • Push objects through respective hatches
  • Press the CLEAR button to clear the cabin of extra debris
  • Repeat

Step 6. Creating the Space

Set design is an incredibly important part of VR, and one where the old tricks of the architectural trade have an entirely new medium of expression. As we mentioned earlier, this was actually the first step in designing VR Cockpit, since it was intended to provide an environment for interactive experimentation.

How you structure your space will depend entirely on how users can interact and explore it. Typically, your space will have foreground, middleground, and background elements. The right balance of set design elements takes an enormous step towards the spine-tingling sensation of presence that characterizes the best VR demos.

Foreground: Objects and interfaces that you can interact with directly. In Weightless, you have the ability to shift the foreground by floating around the space station, in order to reach floating objects. Your hands always occupy the foreground. (Check out this guide to laying out your foreground user interface.)

Middleground: Within the middle distance of the scene, there may be elements that you can bring into the foreground, or which frame the environment. The various interfaces scattered throughout the Weightless space station, and the massive buttresses of the station itself, form the middleground.

Background: Beyond the middleground, the background frames the world where your experience is set. In Weightless, this is the starry universe beyond the station windows, including a prominently placed planet.

Another way to think about space in VR is how it’s experienced at the human scale — in terms of attention, structure, and affordance. As you move your gaze through the scene, where does it land and where do you focus? How does the structure of the space around you make you feel and how does it influence you to move? How do the objects and scene elements you focus on communicate their purposes and statuses?

Step 7. Visual Design

Now that you have your space, it’s time to make it look awesome — with objects, textures, and other set pieces. Character and object animation and design is often one of the biggest challenges for non-designers, but it doesn’t have to be. There are plenty of great guides to 3D modelling and design for Blender and MAYA, or you can dig around on the Unreal and Unity asset stores for model packages.

As Press Bird to Play creator Gerald Terveen put it:

I think I use a different approach from most game designers because I’m not able to create my own models. Instead, I’m working with stock models I purchase. I treat Unity just like a box of Lego — where all the models I already bought get reassembled into the world I create. I have a diversified collection of models from great artists all over the world that fit a fantasy setting similar to the games I played as a kid/teenager.
By combining these assets, I can create my own unique version — for example, the elevators in PBTP use elements from five different artists.

Even if you’re operating on a shoestring budget, it’s possible to build really beautiful worlds and interactions with assets that will carry into future projects. (For example, check out this blog post on 6 kickass Unity assets.)

On the Unreal side, Epic recently unveiled a massive treasure trove of 7,600 art and sound assets from Infinity Blade. That includes stone set pieces, flowing lava, magic effects, smoke, lightning, weapons, and thousands of raw audio files and sound cues.

In other words, a really big box of Lego.

Step 8. Beyond the Interaction Loop

At this point, you’ll also want to consider how users will interact with your virtual world in other ways. Outside of its core interactions, Weightless includes a lot of other little features, including window shutter and music controls. These are scattered throughout the scene and provide rewarding diversions from the main gameplay.

What about moving around within the space? In demos like Soundscape VR and ElementL: Ghost Story, there is zero body movement within the scene. Famously, Sightline: The Chair (which was recently updated with Leap Motion support) uses a gaze-direction mechanic that drives the user further into the experience while staying seated.

In many ways, these demos avoid one of the biggest challenges in VR — the fact that outside of actually getting up and moving around, there is no perfect one-size-fits-all solution to the locomotion challenge. As human beings, we’ve evolved to navigate through the world using our legs, so it takes a creative touch to design an immersive experience where locomotion is achieved in other ways.

Fortunately, there are lots of exciting developments in this space, as many developers are researching lots of tricks and techniques to make locomotion feel good in VR. Cloudhead Games’ Blink VR warps the user around the room, combining a line-of-sight approach with trigger-based inputs. Vehicle-based games like Elite Dangerous, Eve Valkyrie, and our VR Cockpit demo use a solid point of reference around the player to enable easier motion. Whether you’re running on rails, flying like Superman, teleporting between different worlds, or something completely different, there are many creative ways around this challenge.

Step 9. Music and Sound Effects

Sound is an incredibly important element of creating VR with presence, because it allows you to guide and communicate with a player who may be looking in any direction or paying attention to any number of other things. Good sounds also play a huge part in affording how interactive objects work, and giving feedback during the interactions themselves. Even the subtle sound of a button press or an interface springing to life gives the user information on an intuitive level.

Music and sound effects also have the power to set the mood of an experience. For example, VR Cockpit has a deep ambient sound that suggests outer space, while Weightless uses a soft piano track that reinforces its meditative quality.

Check out our post 4 Ways to Unleash the Power of Sound in VR for more sound design tips and resources. To bring sound into your project, check out SoundCloud’s commercial use search and The Free Sound Project.

What will you build for the 3D Jam — and how will you build it? Everyone’s development roadmap is a little different, so we’d love to hear your process for starting a new project. Post about your experiences in the comments!