Reluctant to Hop on the VR Bandwagon?

We’re trying to figure out why, because us too 🤷🏻‍😪

Hey there!

Our names are Ankita and Linda. We are both juniors (almost seniors #yikes) studying Computer Science. We conducted a User Study at Brown University a few weeks ago on Virtual Reality Software and Hardware and the results that we obtained were far too interesting not to share. 😉

Before jumping in, we’d like to make one thing clear; prior to this study, we both had little to no exposure to Virtual Reality. This in part stimulated the creation of our user study. We wanted to understand why we had previously not explored VR and …. why (we hate to admit this) there was a lack of motivation to do so. The questions that we wanted to ask our participants, including ourselves, revolved around a few key ideas:

1. Is VR development frustrating?
2. Are people even mildly aware of the niche applications of VR?
3. How accessible is VR?

These questions served as the basis for creating a round robin 2 hour workshop comprising of 6 activities where small groups of people would dabble with current VR software, hardware, and media while allowing us to gain valuable insight from their feedback, observations, and experiences.

Presenting to You — The Final 6

After much brainstorming, we settled on the following activities:

Activity 1 Understanding the People we are Studying: Participants fill out a questionnaire about their background, interests, and exposure to VR.

Activity 2 Software: Participants complete two software tutorials using Unity and A-Frame.

Activity 3 Hardware: Participants navigate through two different interfaces with little guidance and instruction using the HTC Vive

Activity 4 Troubleshooting: In an effort to evaluate how well VR software is documented on the web, participants are asked to resolve commonly encountered VR bugs.

Activity 5 What are these Applications Used for: Participants are asked to match VR technologies with their applications to evaluate their level of awareness regarding different use cases of VR technology.

Activity 6 Media Lab Experiment: Participants use the Google Cardboard to watch a 7 min. VR clip using the New York Times VR App.

Woah…but how did we get here!

Converting a virtual plan into reality 😏 required a bit of effort and TONS of planning.

We had to think about:

• The questions we wanted to ask in our surveys
• The number of participants for our study
• Where and who to ask about borrowing hardware
• How to market our event to get a diverse pool of participants
• How to run such a large and long workshop
• The list continues… 📝

Thankfully, there were many resources that we were able to make use of to make this workshop possible!

1. We sat in on a thesis presentation that coincidentally discussed about how to conduct a “good” user study. We grilled the presenter afterwards 😁 and were given some tips on how to design our user study, come up with questions, and visualize our data. We found it helpful to have the presenter recollect certain steps he wished he had done better or what he would have wanted to improve in his user study!
2. Ankita audited a class two years ago 😱 that used Google Cardboards. Ankita was able to narrow down the source of these Google Cardboards and much to our surprise, the source was willing to provide headphones, loaner laptops, and phones!! 👌 👌 👌 Huge shoutout to Brown Media Services! 👊🏻
3. Linda had played around with the HTC Vive during Hack@Brown 🔥 — Brown’s annual 24 hour hackathon — and was curious how we could go about using it for our User Study. She was able to track down the right people and arrange to have TWO during the study. Go Linda!

While this is all dandy 🍉 🌈 🌺, we knew from the start that getting people to take two hours out of their hectic schedules would not be easy! Therefore, advertising the event had to start ASAP. We…

• drafted promotional paragraphs to publish in daily newsletters ✅
• encouraged our professors to share the event with their students ✅
• made a Facebook event ✅
• made posts in several Facebook groups ✅
• had our banner displayed on the monitors in our CS building and libraries ✅

We were getting people hyped about Virtual Reality, ohhh yeahhh! 💃🏻

We put as much energy into getting participants into getting volunteers 😍 We partnered with Mosiac+ — a student led initiative at Brown that aims to create an inclusive space for URM students in Computer Science. We were thrilled to see our volunteers get excited about leading activities that would expose people to VR for the first time!

With marketing on full blast 💨, we saw more and more people indicating interest to participate. We selected 30 participants and this is how the spread looked like!

Logistics all good, but what about the content!?

For things to run smoothly, we generated guides for each activity for both participants and activity leaders.

Here is an example of what we provided to the participant and leader for Activity 2. 😎

Andddd another example of what we provided to the participant and leader for Activity 4. 😎

👏We made it! 👏

We broke down our 30 participants into 6 groups of 5 people! We made sure to email all participants with information regarding what group number they were in and reminders such as:

All participants had access to a document with links to complete surveys after each activity. The document also included a timetable with locations for each activity — indeed a busy two hours ahead for our participants 😱

We’d say things went smoothly!

What you’ve been waiting for…

The first step we had to take before analyzing our data was to sort participants by personas.

Personas are ⭐️ important ⭐️ because they help us understand what kinds of interests and backgrounds would lead a participant to have a certain experience with VR. In addition, it would help us quote them on open-ended survey questions or feedback later whenever we needed to support a claim with evidence. We came across many personas in our study!

Results

🎉 🎉 🎉 Activity 1: 🎉 🎉 🎉

We showed you a few pi charts earlier to wrap your head around the kinds of participants that attended our user study, but now we’ll go into a bit more depth.

Have you had any recent injuries? Do you wear glasses?

None of our participants had any recent hand or head injuries or vision impairment.

A majority of our participants did wear glasses, which could speak to how their experience would differ from someone who did not have corrected vision.

Do you play video games often?

Only a few of our participants play video games often. Most said they “hardly” play.

Where do you think you first heard about VR?

Many people heard about VR from an article or video online. Specific sources include:

• Ivy Film Festival’s VR activities
• The original Tron movie in the 1980s 🎬
• Hackathons
• Newsletters

Why might you not have had any VR experience?

“I’ve never had the tools.”

“I don’t play computer games / am not cs-focused.”

“I just don’t know where to start. The costs of acquiring VR headsets and games is a huge factor. I also don’t know what I would need to learn if I were to start developing VR applications / games for my own use.”

It seems like the general perception is that VR is an inaccessible activity that requires you to have a Computer Science background or money to be able to pay for the necessary tools… 💔

What words do you associate with VR?

🎉 🎉 🎉 Activity 2: 🎉 🎉 🎉

Unity

The majority of our participants had never used Unity before and when asked how difficult it was to use, most replied with “very hard.”

Alas, Many Frustrations… 😪

Many participants were frustrated with the GUI of the software.

One participant felt “formatting 3D text is unclear.”

Another participant had to “poke around all the menus to figure out [this] annoying to use and unintuitive, horrible interface” and “had no idea what [they] were doing”

Many felt the inspector properties were unintuitive. They had “very little description explaining what they do” and changing perspective or view angle was not immediately understandable.

About half of our participants “strongly agreed” that Unity felt unnecessarily complex and when asked whether they would see most people picking up Unity and doing an activity quickly, yup, you might have guessed it, the majority of our participants “strongly disagreed.”

While many clearly disliked the Unity interface, one participant remarked how it was “easy once you figured out which menu to use” and how it was “fun playing around with visual features…[and that they could] build up something from scratch” if they wanted to. Most people also responded that they would use the software on any given day and would indeed use it again! 👐 Those that would not use it again had valid reasons too. They either “[did not] have a use for it” or simply “[did] not [find it] interesting.”

Our activity leaders identified the onboarding process for Unity to be a huge pain point among participants — it was long, tedious, and entailed filling out a long profile just to sign up. 👵 They also observed most participants clicking around by trial and error to get the effects they wanted. 😧

In conclusion, Unity’s GUI is both a source of heartache and happiness. Once you learn how to use it, people seem to feel invincible with the tools at their disposal⚡️, but to those unfamiliar with the software, getting a simple “Hello World” to show up is frustrating and discouraging. 😢 A more intuitive and beginner friendly interface could go a longgg way.

A-Frame

The majority of our participants had never used A-Frame before and most agreed that it was difficult to use.

Participants who disliked the platform, without any background in HTML, felt like it was “exclusively built for coders.”

Participants also noted that the documentation was unhelpful. 👎

People seemed to have liked Unity more since it had “some level of physical manipulation…instead of having to look through code to reposition objects.” People found it “weird to use only code to animate entire scenes or programs.”

Most people said they would not want to use A-Frame again 😔 😔 😔 and did not see anyone doing an activity on A-Frame quickly.

Participants with a CS background, however, appreciated how it served as an “extensible framework.”

Our activity leaders observed people “staring and squinting at their screens” most likely due to the poor documentation and difficulty people had grappling with HTML in a short span of time. One volunteer noted that participants were discouraged about the aspect of coding and how it felt “unclear what attributes [in HTML tags] to use or even what the attributes were.” ☹️

🎉 🎉 🎉 Activity 3: 🎉 🎉 🎉

The majority of our participants had never used or heard about the HTC Vive.

To recap, participants were given about 10 minutes to complete two activities using the HTC Vive — one entailed finding Brown University using Google Earth and the second entailed navigating to The Lab to play Longbow.

Most of our participants were familiar with similar handheld devices like the Wii and felt that the headset was comfortable. Those who did not find it comfortable said they were “not sure how [they] would feel if [they were to] wear it for an hour,” describing how it caused them to sweat 😓 as it was “heavy for [their] head” and “bulky.”

People ran into the following frustrations:

• Certain parts of the screen were blurry which made reading text challenging 🤦🏻‍
• Figuring out how to scroll and navigate 🤔
• Unfamiliarity with the buttons and their purpose on the controllers 🤔 🤔
• Scrolling past where people wanted to click — one participant described this as “VERY annoying” 😠
• Being unable to see your hands #spooky 🙄
• The buttons map to the controls differently for each app which made users have to relearn which buttons map to which controls when entering a new game 😣
• Coordinating two controllers 😳

Our participants were able to overcome many of the above mentioned hurdles to ultimately enjoy their experience using the HTC Vive. Many loved how they were able to have a fully immersive experience with realistic graphics and seamless visualization.

Our activity leaders felt that participants found the device hard to use. They also felt that the task was decently mentally demanding while the majority of our participants felt that it was very mentally demanding. Another contradiction that arose was that our participants felt very safe using the device and our activity leaders believed it would be ⚠️unsafe⚠️ for participants to use the device if they looked away.

In conclusion, it seems like people would have had a better time with the HTC Vive if the navigation was more intuitive and the function of buttons on the controllers were more consistent. Although, through trial and error, most people were able to pick up the necessary HTC Vive skills to be successful! 👊🏻

🎉 🎉 🎉 Activity 4: 🎉 🎉 🎉

The sentiment after this activity was definitely not surprising, almost quite what we expected. I mean who enjoys debugging for fun!? 🤷🏻‍

The above word cloud represents what participants described their experience resolving commonly encountered bugs in VR software development.

Most people found the activity difficult to complete.

One participant — with a background in computer science — “searched for the keywords of the problems on Google and clicked on a few links to see what other users have said.” Such a strategy, although straight forward, is an acquired skill that comes with familiarity with debugging. On the other hand, a participant that did not have much programming experience found the activity difficult because she “did not understand the questions or terminology in the responses online.” Perhaps this suggests that current VR documentation guarantees a steep learning curve 🤸🏽‍ and is inaccessible for users who are new to software development. While this may be the case, several participants also found the documentation to be nonexistent and unhelpful. Here’s what a few participants had to say about this…

“Some of the problems appeared to be open bugs with no good solution.”

“Yes because a lot of people posted the same question/issue on forum sites and not many people had solutions or there were lots of different answers.”

“Not really, I was able to readily find users who had the same problem, but not all questions had answers that worked.”

Participants would like to see better documentation; “…documentation [that is] more beginner-friendly” in addition to “…more user help communities.”

To quite bluntly put it, around 77% of participants could not find the answers they needed to resolve the problems we gave them…This is quite disappointing. 🙁

We think there is so much to take away from this activity. Although we felt bad for having participants debug (please forgive us participants! 🙏), the insight that they provided was extremely valuable. VR software development documentation sucks and needs to improve. While there definitely do exist the brave developers that will fiercely bang their head on bugs for days without much guidance or support, majority of people do not have that kind of time or patience. 😑

~ An interesting additional note, our activity leader felt that this activity was not so challenging for participants (while in truth, we know most had a really tough time 👆). This introduces a rather noteworthy contradiction; maybe VR documentation developers also think that their users are not having as challenging of a time resolving their bugs…hmm… 🤔

🎉 🎉 🎉 Activity 5: 🎉 🎉 🎉

We are no market strategists, but we inherently believe that if you are selling a product, its purpose should be crystal clear 🔮. Our results from this activity do not seem to suggest that VR technologies have hit this level of clarity. For instance, after the completion of the activity, one user wrote that she “still [couldn’t] really tell the difference.” Our activity leaders also mentioned that there “was [a lot of] confusion between Blender and Unity.” Despite this, after participants were able to educate themselves on the various applications with a bit of Googling, the below word cloud represents what participants associate with VR:

One participant, with prior VR experience, noted that she “.. did not know Blender could be used for art!”

We also found it interesting that a participant suggested that it would be beneficial to “[add] things to the logo” to make the purpose of certain VR technologies more clear… we can’t disagree, I mean we’re also not sure what to take away from this:

🎉 🎉 🎉 Activity 6: 🎉 🎉 🎉

The Google Cardboard is a wonderful accessory that has helped make VR more accessible with its low cost and accessibility. To put this in context, it was pretty easy for us to obtain 30 Google Cardboards for this study!

Roughly half of our participants were unaware of the NY Times VR app. After users watched a short clip using the NY Times App using the Google Cardboard, it seems like a lot of participants felt sick and uncomfortable. 🤢

One user said her “nose [was] super uncomfortable” and another said she felt “slightly dizzy; weird to lose sense of body in space.” 😵

Words that were associated with VR after the completion of the activity extended to:

A lot of the feedback from our participants entailed critiques on the user interface of the app and the discomfort associated with wearing something on your face and being completely free to move around.

“The user interface is pretty simple and effective, but I could feel my arms fatiguing at the end of the six-minute video. I’d love a head-mounted display so that I don’t have to constantly hold it. That aspect does detract a little from the experience.”

“The app was difficult to search; rather than being a scroll-dash, there should be a menu page with categories and a search bar.”

“Not having to hold up the headset myself, menu navigation while in VR instead of having to take the phone out and put it back in, adjustable focal length for the lenses”

One of our activity leaders also noted that “people’s text messages kept popping in which was annoying for users [who sometimes had to] start from [the] beginning” 😤

What‘s Next?

First of all, if you made it this far, THANK YOU! 🙌 We appreciate you reading this!

If you’re curious what’s in store for us…

We’ve been fortunate to have the folks at RIVR reach out to us to organize something for the greater community. We’ve been in touch with them and plan to host another workshop on VR this coming September; one that we hope will involve less material that is assessed but is rather taught in more greater depth. 💡

We definitely learned a lot through the organization, setup, and evaluation of our user study and felt it would be neat to share! We’ll be back soon with more updates on our journey to understand the potential bottlenecks that exist in getting people more involved with VR #staytuned 🌼 🌼 🌼