Conducting UX Research at a Conference
How the VMware Design team led research with over 1,000 users in 4 days
Just two weeks ago, over the course of 4 days, the VMware Design team ran a variety of focus groups, walkthroughs, and participatory design sessions with 1,134 users. As a researcher in the B2B space, this is something I never really thought possible, but this is the fourth year that the VMware Design Team has pulled it off, and attendance only keeps growing.
VMworld Design Studio
We call these sessions Design Studio collectively, and they started out as an informal way for designers to get feedback from users by taking advantage of the vast number of partners and customers who attend VMworld every year. In its first year, designers managed to talk to 80 users, but now it has grown into an official VMworld event that many of our users look forward to attending.
For those who aren’t familiar with VMworld, it’s VMware’s biggest conference of the year, and one of the biggest for the IT industry, with over 20,000 attendees this year from across the country and around the world. And that was just for VMworld US, which took place at the Moscone Center in San Francisco this year. Part 2 will be held in November in Barcelona as a more accessible option for partners and customers on the other side of the pond. It’s an event for people to learn about VMware and the landscape of cloud computing technology. Attendees find out what’s next for VMware and what the company is doing to leverage up-and-coming technology and improve established product offerings.
Planning It
Design Studio is a big part of the VMworld experience, as it gives users the opportunity to help shape the design of the products they use. Designers and researchers for all of VMware’s products start planning months in advance. We work collaboratively with our PM and engineering teams to decide what topics to bring to VMworld. Together, we seek to answer questions like: What are the top issues we need to solve? How can we get the most value out of a focus group-type setting?
Behind the scenes, our Design Program Manager, Afifa Tawil, works with VMworld organizers to plan the logistics. One of the most important tasks is to find and set up a space that can effectively run up to a couple dozen focus groups simultaneously. This year, Design Studio took place in the Yerba Buena Center for the Arts, just steps away from Moscone, and we had the whole Forum to ourselves. Overall, 421 sessions covering 60 different topics were run over the course of those 4 days, with over 100 designers, researchers, program managers, and product managers participating.
Part of the planning effort also involves ensuring that our Design Studios show up in VMworld’s content catalog — that way, attendees can sign up for the sessions that interest them and we don’t have to worry about self-recruiting. At this point, Design Studio has built up enough credibility, both internally and externally, that it’s no longer a question of whether research will happen at VMworld — just where and how. So if you’re a researcher or designer fighting to create a space for research at a conference, start small and persevere; eventually your results will speak for themselves.
Design Studio for us never stops — besides the mountain of insights we gain from conducting research at this scale, one of the main reasons we do it at VMworld is to invite users to engage with us throughout the year. At any given time, we have at least 5 online, remote Design Studio sessions underway. However, these are more for 1:1 discussions and research methods that work best in those settings, such as interviews and usability testing.
Design Studio Underway
VMworld Design Studio takes advantage of the collaborative, discussion-oriented nature of a focus group setting. Most sessions had at least 3 attendees, though many had up to 8. A lot of users seem to like this setup even more than our remote 1:1 sessions because they’re guaranteed to learn something about how other IT professionals do similar things. My sessions were focused on API usage, but I had a couple of participants who didn’t use APIs at all and came instead to learn from other participants with more experience.
Sessions ranged from focus groups to participatory design sessions to prototype walkthroughs…
- Focus groups: one side of a spectrum, in which sessions are almost entirely discussion-based — designers and researchers who chose this method were largely focused on strategic or foundational insight. For example, I was trying to understand users’ processes for discovering and consuming APIs, agnostic of the APIs they use.
- Walkthroughs: the other side of the spectrum, in which everything is based on a design — usually a proposal for a redesign of some feature or product. Because of the group setting, it’s not a reliable way to gain usability insight. However, it does offer designers an excellent forum for real users to critique their designs and for them to learn more about the organizational context in which their designs are used.
- Participatory design: a happy medium between pure discussion and design-based activities. Many designers combined focus-group-type discussion with design walkthroughs and included some sort of collaborative activity, like sketching what an ideal billing dashboard might look like to them and then comparing it to a real prototype. This was a great way for designers and researchers to understand users’ mental models.
Getting creative with methods
Let’s take a look at some of the ways the VMware Design team made their sessions interactive and engaging while also capturing valuable research insight…
Build your own IAM policy
Grace Noh wanted to understand how her team can upgrade Identity and Access Management (IAM) capabilities for VMware Cloud products so that they scale to complex scenarios and provide security and system administrators with with the granular control they need. To do this, she needed to rethink VMware Cloud’s model for creating permissions policies, so she designed an activity in which participants built diagrams to show how they would create policies at their companies. Participants were presented with different scenarios and cards representing different identities (e.g., the networking team), roles (e.g., write access), and resources (e.g., clusters). They then collaborated in pairs or groups to build a policy model that made the most sense to them.
Interactive Journey Mapping
Priyanka Shetye, Deneva Goins, and Harshath J.R. aimed to better understand user journeys related to the vSphere lifecycle — what does the upgrade process look like from the user’s perspective? They designed physical journey map templates, broken down by different phases — “Before” (the upgrade), “During,” and “After” — and categories — “Products,” “Resources,” “People” — and provided stickers representing personas and tools they thought might be involved. This was all to simplify the activity for participants and give it more structure. Participants would take turns discussing their challenges at each step of the process. It turned out the upgrade process was even more complex than the team had expected, especially because of third-party dependencies and compatibility issues.
Understanding the off-screen experience
Alex Zhu focuses on the onboarding experience for VMware Cloud products. He wants to ensure that the on-screen process for onboarding to VMC makes sense when off-screen factors are taken into account. To this end, he demonstrated the current onboarding flow to his participants and asked for their feedback. Afterwards, he presented them with 4 paper cards representing that flow, as well as cards with other actions or tasks he hypothesized they might engage in, like holding a meeting with their procurement team. They were asked to arrange the cards in such a way that they represented what the onboarding process would look like at their organizations. Participants noted who would be involved at each step of the process with persona cards and wrote their own card when an important step was missing. Alex stressed the importance of understanding end-to-end experiences when designing a service — what we design has an impact beyond the on-screen UI.
Overcoming Challenges
If you’ve ever conducted a focus group before, or even just a user interview, you’ll know that no amount of prep work can fully prepare you for everything. This is especially true when it comes to VMworld, since we can’t control who shows up to Design Studio (beyond catering our session titles and descriptions to a target audience).
Oftentimes, we’ll get people who are just interested in learning about a product, but don’t actually use it. In that case, we’d have to tweak the session to make it less about design feedback and more learning about their needs and why they’re interested in the product. Other times, as was the case with my session, we’d get a mix of very different personas, which can mean the script you spent hours preparing goes out the window.
Research is all about dealing with the unexpected, ambiguity, uncertainty… but that’s also where it draws the most value. It makes us re-evaluate our expectations, make sense out of ambiguity, and move forward with the confidence that we are making the right design decisions.
Impact
When you’re busy juggling stories and feedback from dozens of users, it can be difficult to tell what the impact of your research will be, so it’s always encouraging to see partners and customers reacting positively to Design Studio.
It emphasizes the importance of taking the time to not just listen to your users, but also to collaborate with them to effect changes in product design that will improve their workflows. It’s a continuous process for sure, as there are always improvements to be made, and the regular Design Studio sessions we hold throughout the year are testament to that. However, it’s only twice a year that thousands of IT professionals who all interact with VMware products in some way come together in one place. It’s also a guaranteed opportunity for product leadership to see designers and researchers at work, which is crucial for building internal credibility. Many of us had PMs and engineers sit in on and engage in our sessions. It’s an excellent opportunity for them to get face time with users as well, and they see that as invaluable.
Next Steps
We’re in the process of synthesizing all the data collected and will present the results of our research in the next couple of weeks. We can already see some change coming about — those design feedback sessions are leading directly to swift changes in product design, while the more foundational sessions, even though final results aren’t out yet, have already contributed to a new understanding of our users. Our research will undoubtedly lead to long-term strategic impact, especially because every Design Studio is a company-wide effort that grows every year. Results will be presented to executive leadership in the coming weeks, and we can see priorities beginning to evolve. Many of us will be following up soon to chat with the 631 new partners and customers who signed up for Design Studio this year.
In less than 2 months, our fellow designers in Sofia, Yerevan, and Bangalore will be leading another round of in-person Design Studios at VMworld Barcelona, expanding upon the insights collected at VMworld US. We hope to see you there!
We’re hiring!
The VMware Design team is looking for talented designers to help us continue transforming enterprise design. Check out our open positions!
