10 Lessons: Building GitHub’s Early User Experience Research Practice
The past few days I’ve been thinking a lot about what a team of two was able to accomplish in three years at GitHub, spanning some pretty difficult times for the company. It’s a pleasure to look back and even to see what I overlooked and some of the challenges I personally faced.
What follows are 10 lessons I learned while building GitHub’s early research practice. To be sure, I didn’t practice all 10 perfectly all of the time, in fact, some lessons are aspirational. In spite of some bumbling, I also experienced an incredible amount of learning.
Truth of table stakes
The one lesson I didn’t include as a bullet point is the importance of having confidence, self awareness, and a growth mindset. Easier written than mastered. While I find it natural to have an abundance of confidence in research findings and the process of how we analyze our data, I also find it challenging to have that same level of confidence in relationships, intuition for what to do in difficult human situations, and humility in asking for help sooner rather than later.
My Top 10
10.) Prioritize privacy. Put strong practices in place early to protect user data, especially self-reported survey data. Privacy will become an important part of your research team’s brand.
9.) Host research reviews. Synchronous discussion/presentation sessions with team members from across departments will help you communicate research results by any means possible. However, a word of caution (insert smile here), sometimes you’re going to be asked for word clouds (recommend bigrams).
8.) Practice research rigor. Always review (triple check), clean the data, and purge when necessary (avoid Garbage in, Garbage out).
7.) Start with “Who.” Begin analysis by taking a critical look at who participated in your research (e.g. surveys, interviews, randomized controlled experiments). Are some groups overrepresented, while others are underrepresented? Why?
Who will help you better understand your research’s blind spots. Think about those blind spots and how/if you can fill them in, so you can depict a more complete analysis/story.
6.) Build for enablement. You can’t give empathy away, so design a practice that helps engineers and designers stand up their own research studies. This can be as simple as creating a database with a lightweight UI that makes recruiting participants who have volunteered for research easy, as well as define a set of reusable protocols.
Enablement will also free the research team up to do other types of work. Do this sooner rather than later, before you’re so deep in studies that you’re building the boat in the middle of the ocean without land in sight.
5.) Encourage graph literacy. Graphs help tell the story. Show your colleagues how to read a graph and see what is obvious, but then challenge them to cover that up and look for what is interesting.
4.) Look for & hire generous people. I love working with people who embody a spirit of generosity. These folks are often willing to reach over and into hard-to-reach places and teach their peers as we build a better workflow, product, and company together.
3.) Say “No.” If you say “Yes” to everything, you’ll find that you’re saying “No” to many things by taking on too much and not doing an outstanding job on all of the things.
2.) Teach technique. Write blog posts that explain research methodologies, and weave in real-world examples of your team’s studies. This will help both the skeptics and the curious across the organization come to see research as a technical field and it will improve their trust in your results.
1.) Prioritize relationships over research rigor. Rigor will fail you every time if you don’t have good relationships in place with product managers, product designers, engineers, and marketing and sales team members.
This is tricky, because we’re human there will likely be a few bad experiences and sometimes consecutive bad days. Resist writing relationships off. It’s probably likely that you too made some poor decisions along the way, but you learned and grew, and that’s what will happen with people around you and you can help.
Each of these 10 lessons helps to surface what’s meaningful in data, how to give data a voice, which then helps teams act on it. I truly believe the best experience a researcher has is when she learns something brand new about the world, not alone, but rather alongside her team members.