open door user testing
the importance of team involvement while testing
When a digital product launches, one hopes it’s met with user elation, high retention rates, and feedback that lets you sleep at night. The following story is about my experience when this does not happen and the critical role user testing can play in unifying a team to react and rise to make the hard decisions necessary to improve the product.
During my first week as a Product Designer at Codecademy, we launched a new product that was the first step toward teaching people skills they could use to get closer to a web development job. The fact that it was a first step showed in the post-launch metrics, with higher than expected drop off and low engagement for key interactive environments throughout the course. It was hard to sleep at night.
We had metrics which told us there were problems, but they didn’t provide enough insight to justify a major product pivot. We required the “why” for the drop off rates — was it due to the visual design, content, or interactions?
The team had varied opinions on what to do, but there was growing concern that something needed to change. We were faced with a critical product decision: did we have enough information, understanding, and empathy to change the product and improve user retention? Or did we need to go back to the drawing board, starting with user testing on our launched product?
We concluded there were more questions than answers and we needed to go back to our users for future direction. The team had done some pre-launch user testing via usertesting.com, but unfortunately not all the observations were acted upon before launch.
Herein lies one of the most common issues I’ve run into around user testing— lack of documentation and support across the team. If user testing was going to be our main tool for moving forward with this product, we had to be sure it involved the entire team and that everyone was behind it.
This meant that user testing had to result in a clear understanding of the problem, but most importantly, it had to be the primary instrument for unifying the team under a common direction. Until Codecademy, I had not seen how important this second factor can be and while not revolutionary, it’s a small change in the process that can have a massive impact. This is what I refer to as open door user testing — open the door to the entire team, not just researchers and designers, but everyone — the benefits are deep and long lasting. Below are the key principles that myself along with my teammate Lalith Polepeddi, introduced to our team during this process:
1. Keep the door open
We set up the user testing in two rooms and hacked the rooms together using Apple Airplay, landlines, and Silverback. Testing took place in one room, while the other had an open door to any team member who wanted to observe and take notes. The key factor is participation — every team member that sits in on a testing session, is one more that gets to share in a unified understanding of what must change in the product.
2. Tell everyone
You don’t have to be a designer or researcher to observe a user testing session. Yes, it’s critical that the person conducting the test is well versed in this usability practice, but the more observations you get from varied sources, the better. We encouraged participation by emailing everyone in our company with a schedule of the sessions, and let them know in team meetings that we wanted their feedback during each session.
3. Close the door and document
As we learned before this process, without documentation and exposure, user testing quickly loses it’s impact. Good documentation happens before the first session begins, by telling all team members who join exactly what is expected of them. Once the session is over, we close the door and go around the room to register each team member’s observations on a whiteboard.
At the end of the small batch of user testing we had half of our entire company involved (yes we’re only 25, but still!) This made for stronger clarity on the problems we set out to solve and a shared agreement on the priority of features amongst developers, designers, founders.
At Codecademy, user testing is becoming an integral tool in our product development and we look forward to sharing more stories and getting feedback on our process!
Appendix: Setting up your open door testing session
- Make your plan: A good place to start is Google Ventures’ resources and guide to creating your plan.
- Source and screen users: We did this through Craigslist and used Google Forms to screen users to match our personas.
- Hack together two rooms: We recommend Silverback for capturing your screen, a landline for tying audio, and Apple AirPlay for sharing the screen with the other room where team members are observing.
- Define team involvement: We communicated the schedule and what was expected of anyone who wanted to sit in on a session both via email and verbally during the week leading up.
- Document: We kept everyone who participated in the room until we had documented all observations — from here it was up to us, the product team, to document and archive the findings.
- Share the findings and process: We presented the key findings and an overview of the process and sent out a summary to the entire company.
