“No causation without manipulation.” –Paul Holland
At GitHub, we frequently use experimental research designs to study the effects of new features and models. This post describes when and how we use experiments, and why they are such a powerful tool.
tl;dr: experimental designs are the most efficient means of establishing causal relationships, but causal inference is hard!
To discuss randomized controlled experiments, in this post, we’ll use the example of the Golden Ticket experiment. …
There’s an infamous story and that often comes up in product debates to help justify not listening or reacting to customer requests. It’s inaccurately attributed to Henry Ford:
“If I had asked people what they wanted, they would have said faster horses.”
Customer requests are an important input in product development and the role of research is to listen to and surface the how, why, when, and where, behind them. We’re looking for patterns, use cases, and the human motivations and goals driving the conversation.
At GitHub, in 2015, customers told us they needed a complex feature called “branch permissions.”…
The past few days I’ve been thinking a lot about what a team of two was able to accomplish in three years at GitHub, spanning some pretty difficult times for the company. It’s a pleasure to look back and even to see what I overlooked and some of the challenges I personally faced.
What follows are 10 lessons I learned while building GitHub’s early research practice. To be sure, I didn’t practice all 10 perfectly all of the time, in fact, some lessons are aspirational. In spite of some bumbling, I also experienced an incredible amount of learning.
We recently launched the second wave of a study internally named, The New Account Creator Longitudinal Study or “Project NAC.” We’ve been following roughly 3,200 new users since September 2015, checking in periodically to ask:
Longitudinal studies are complex and, at times, difficult to complete. A person in the cohort may respond…
I truly hate using word clouds to present open text data, which is a research lovers pot of gold. Word clouds –well, I feel like they cheapen the richness and elegance of open text. However, every once in a while, a project comes along where word clouds make just enough sense to use as a visualization tool –they are especially accessible for diverse audiences across an organization.
As part of the annual GitHub Tools & Workflows survey we asked an optional open-ended question at the end with a single open-text field:
How would you describe GitHub in three words?
GitHub.com’s business model charges for private repositories with two types of plans –personal and organization. In 2015, we launched a randomized controlled experiment lovingly named “The Golden Ticket,” giving coupons for free private repositories to 39,800 people.
What follows recounts some of the high-level insights we drew from this experiment that might help you research your product’s value or price.
We used a classic experimental design with 40k tenured users on free accounts assigned randomly to treatment and control…
With research we’re always on the lookout for patterns–some are obvious, but the ones we care most about are interesting. However, obvious and interesting don’t usually fall into the same group. This post will cover an obvious insight, an interesting insight, and a technique to help you rethink your graphs.
In 2013, we conducted 20 field interviews with new users and realized that there was an increasingly gap in newcomer git knowledge–on the qualitative side it looked like more people were signing up for GitHub without much or any git knowledge. …
GitHub began as a tool for hobbyists and evolved with organizations and on-premise enterprise products into a tool for professionals.
We’re going to share 9 key insights from GitHub’s annual Tools & Workflows survey, welcoming debate about “learn git v. less git,” text editor and IDE popularity, and shining a light onto some blind spots.
One finding, in particular, “Professionals solve their problems with support from human peers” surfaces a critical discovery about human behavior: the distinguishing factor in solving challenges getting started with Git and GitHub is that professionals have greater access to people who can help them solve…
Most web services experience seasonal lift in new user sign-ups and traffic. This is especially true in January when people return to work and school, and resolve to make changes to behaviors. Specifically, many new users arrive to apps with their highest potential to learn something, do something. However, new users readiness is often quickly lost when your product’s experience doesn’t present them a compelling path forward.
RealTalk™ … Over the past three years as our newest users arrived to GitHub, who were different than our early adopters, we failed to engage the majority and lost them.
This large leak…
Q: If we could have done one thing to improve your experience what would it be?
End-of-year (EOY) is an excellent time for a retrospective on your app’s growth –specifically looking at where you didn’t grow. At GitHub we conduct a short annual survey, dubbed “The GitHub 365.” We use the 365 to examine what happened with people who signed up for an account, but at some point ceased to return.
Let’s talk about your product:
UX Researcher. Today: Googler on Firebase. My exs: Airbnb (2018), GitHub (2016) & Mozilla (2012). Human interface to Paisley pug.