How to Design VR…in VR.

Last weekend two totally awesome guys named Shiv Kumar and Nate Black asked if we should sketch our new VR app concept in VR and my response was,” Dude, yes!”. Here is the end result:

One step in an interaction flow

Let me back up. My Udacity VR Developer course is running a Ready Player One style #VRjam / hackathon. The three of us got paired on a team, met in SoMa San Francisco and ran a lightening design session:

Lightening round of white boarding

Shiv and Nate pitched a Music app that let’s you see and experience sound as a physical object. The idea sounded STELLAR (yes, all caps) so we started by asking ourselves a philosophical question…

Utility or Art?

Should what we make be a solution for a lot of people? Should we design something that solves a pain point so intense that people open their wallets immediately? Or would we design the whole enchilada from something that we enjoy?!?

Originally we started by drawing a user journey of how a user might progress through the experience but then realized common UX methodologies aren’t always the right fit for creating and refining ideas. Templates are great but oft are born when new paths are forged. Shiv and Nate came up with the idea of playing a Bon Iver song ‘Holocene’ beginning to end and then we would whiteboard what came to mind. So we let the song become the user journey. Yea, we decided to Music Board!

As the song ended we took turns explaining what we had envisoned. Below you can see Nate (right) expanding on Shiv’s ideas (left).

Shiv and Nate doing a deep dive

Here are the ideas we came up with:

Orbiting: Music orbs act like planets that you can control.

Conducting You have a symphony that you conduct.

Uncovering You explore a dark world that gets illuminated by musical objects.

Below are some rough sketches that just seemed to flow from Nate and Shiv

A few microinteraction concepts
After I showed Shiv and Nate the slides Shiv looks at me and said,” Wanna mock it up in Tiltbrush?” My jaw dropped!

Basic Concept Sketches

The next step was to sketch what each step might look like at a high level.

User logs in, sees music as an object and throws it into orbit.
User is pulling an object out to change the sound and smashes in into 2d configuration

These basic concept sketches allowed us to ideate about what we do and don’t like and make conceptual changes that cost us almost not time. The slides took all of 20 minutes to make!

Sketching in Tiltbrush

After I showed Shiv and Nate the slides Shiv looks at me and says,” Wanna mock it up in Tiltbrush?” My jaw dropped! I said,” How would we even do that?” Shiv gave me a crash course and said,”Have at it bro!”

So I did!

Prototyping in VR

So the great thing about doing this is you can actually export the models as FBX and drop them right in Unity so not only are you evaluating what you do and don’t like but once you have something you like… it is already modeled! The downside was I wasn’t aware of a copy paste so I had to recreate an instance of each asset.

Ultimately I decided to make 4 states of the experiences very similar to the google slide screen grab I posted above.

Below is the final product of these activities:

User discovers a part of SoundVerse

The idea is that the user walks up to one instrument (of many) and it ignites in this phosphorus glow that beckons them to interact with it. The other orbs/spheres are diminutive and don’t call the user’s attention. They will eventually whether through gaze, proximity or activation order in the song sequence. We have yet to determine that interaction.

But I think you can see the value of this rapid prototyping technique!


My conclusion is that I have no conclusion. AR/VR is the wild wild west. There are few best practices and everything is a giant experimentation. This was my first time experiencing this tool and approach. It was very effective. My next step is to prototype in Tiltbrush more often as well as look into other tools in VR that can help with this mission. It was an incredible experience and I’m excited to wrap up our concept app for the VRJam contest!

Next Steps

Usually this part is reserved for me to tell you what I will do next but I’d like to suggest something you should do! Go watch the Bon Iver video we talked about earlier:

You will be hearing from Nate Black, Shiv Kumar and I real soon! Stay amped!