User Research on a Massive Scale 🔬
How we talked to almost 500 users in 4 days at the VMworld Design Studio
Setting up user research and usability sessions is hard. The logistics alone of finding the right participants and scheduling times that work for everyone can be a nightmare. As designers, this is often a blocker to truly understanding our users, especially in the enterprise design world where users who are technology specialists or IT administrators are hard to come by.
But two weeks ago, our design team at VMware talked to 482 users over the course of 4 days. How was this possible? 🤔
VMworld Design Studio
Two weeks ago was also VMworld, VMware’s biggest conference of the year with over 21,000 attendees across the IT industry from around the world. The conference consists of various keynotes, breakout sessions, and activities about VMware products and cloud computing technology.
This was the third time our design team organized something called the VMworld Design Studio, which is a series of small-group sessions with customers on numerous products and topics. We tested early design concepts and ran engaging research activities to gather valuable user feedback from VMworld participants.
Recruiting participants
We recruited most of our participants through the VMworld conference catalog where attendees can sign up for sessions that they are interested in and add them to their schedule.
However, some teams handpicked their participants through alternative means such as tapping into existing customer channels or reaching out to those who have participated in our research studies in the past. This is where maintaining good relationships with the people who have previously participated in your studies pays off. Not only will they be more willing to participate, but they can also see the product evolve as time goes by.
💒 User “Proposals” 💒
Did I mention that these user sessions were held in a wedding chapel in Las Vegas, NV? We set up tables inside the wedding chapels at the Mandalay Bay Convention Center where the conference was being held. This is where we met with users, listened to their feedback, and offered them cupcakes with hearts on them. 💘
Focus groups
Typically, each session had an average of 2–3 participants. It was a focus-group-like setting where users could discuss together, disagree with one another, and learn from each other. It isn’t often that IT administrators from different companies meet each other — but when they do, they learn a lot about how each company runs their data centers differently.
In this focus-group setting, we believed it was important to provide a safe space to let each participant have their own voice. So, we would constantly give reminders in each session that every person has unique experiences and encourage them to vocalize their own perspectives.
Session format
Sessions were 1-hour long and facilitated by 2 designers or researchers on average. The majority of the session would be spent doing a research activity or walking through usability exercises. Each session used alternative research methods depending on the goals of each study, some of which are shared below.
At the end, each participant filled out a questionnaire that asked about their background, asked questions pertaining to the session topic, and provided areas where they were free to write down additional feedback or leave us any “proposals”. 💍
Hands-on activities
One of the most prevalent areas of feedback was that participants loved how hands-on the sessions were. Here are some examples of the interactive, hands-on sessions that we did:
Choose a persona — any persona!
One of our designers, Janet Amaro, organized a persona card-sorting activity. She made “playing cards” that represented various personas and tasks, and asked participants to match persona cards to task cards. This is how she started to draw the lines between who manages what, who is responsible for what, and who is who within an organization.
What features are you willing to invest in 💵 ?
In their session, some of our designers and researchers Anna-Marie Panlilio, Yu Xin, and Patrick Davis handed out fake money to participants and asked them to invest in features they want. Each participant received $600 to distribute however they wanted among a list of possible features. This activity was not only fun and engaging for the participants, but also helped the team determine the prioritization, importance, and weight of each feature.
Navigating mental models ⛵️
Another session ran by Varsha Jagdale and Peter Shepherd was also a card-sorting activity, but aimed to understand mental models for navigation. They asked participants to organize navigation items into groups, and learned that they resonated more with task-based organization (Set up, Manage) rather than object-based organization (Resources, Bindings). What they realized was that asking participants to talk through their thought process while sorting cards revealed a lot more information than just the card-sorting itself.
Building internal credibility
In order to make the Design Studio successful, we had to coordinate with many different people and teams across VMware. My design lead, Kevin McBride, essentially orchestrated the entire Design Studio and worked closely with multiple teams to lock down all of the logistics of the sessions.
Many product managers and engineers from our product teams stopped by the Design Studio to sit in some sessions. It was a great opportunity for these stakeholders to hear first-hand from our customers what they like or don’t like about the product.
We even had VPs, CTOs, and other high-profile individuals that work at VMware come by to check out the Design Studio. They were amazed at the results and now see the Design Studio as an integral part of the VMworld conference.
Having all of these types of people exposed to what we do as a design team builds internal credibility and demonstrates the importance of design as a company-level priority.
Tangible impact
When designing for enterprise, sometimes it’s hard to see the direct impact that your designs have on people. At these sessions, we heard from customers that worked in a variety of industries such as healthcare, military, finance, education, technology, and more.
We learned how critical data protection and security is to customers within the healthcare industry. We learned how data centers in universities are smaller and often segmented by department. We learned how oil companies that have remote oil rig locations need individual servers at each edge location for reliable connectivity.
Hearing these tangible stories of how VMware’s products have an impact on the real world helped our design team to realize how valuable the work we do is to various industries.
Empowering users
Out of 800+ different sessions at VMworld, some attendees found the Design Studio sessions to be the most rewarding:
“This is by far the best hour I spent at VMworld so far.”
“I liked being involved in the design
process of a future VMware product.”— VMworld U.S. Attendees
And when asked what they’re favorite thing at VMworld was…
Overall, this large-scale user research initiative is not only rewarding to our team, but it empowers our users and customers to have a voice and reminds them that they can influence the user experience of VMware products. Of course, we conduct regular user research outside of VMworld, but the Design Studio enabled us to perform research on an unbelievable scale.
If your company happens to host or attend large conferences such as this one, I recommend taking advantage of events like this by coordinating your own “Design Studio”. Especially in enterprise, it isn’t often that dense populations of administrators, technologists, and other enterprise-type users all convene together in a single place.
VMworld Europe is happening this November in Barcelona, Spain. And you guessed it, we’ll be holding another Design Studio there!
We’re hiring!
The VMware Design team is looking for talented designers to help us continue transforming enterprise design. Check out our open positions!
