Product Pricing, Attitudinal Data, & GitHub
“However small the chance might be of striking lucky, the chance was there.”
― Roald Dahl, Charlie and the Chocolate Factory
GitHub.com’s business model charges for private repositories with two types of plans –personal and organization. In 2015, we launched a randomized controlled experiment lovingly named “The Golden Ticket,” giving coupons for free private repositories to 39,800 people.
What follows recounts some of the high-level insights we drew from this experiment that might help you research your product’s value or price.
Experiment design
We used a classic experimental design with 40k tenured users on free accounts assigned randomly to treatment and control groups. The treatment group was divided randomly into three subgroups:
- 6,600 people — received a coupon for 1 free repository
- 6,600 people — received a coupon for 3 free repositories
- 6,600–received a coupon for 5 free repositories
In addition to measuring activity (redeeming the coupon and creating private repositories), there were two other components to the study:
1. Screener data, which collected from coupon redeemers (4,418 participants).
2. Exit survey data, which was sent to all members of the treatment group (2,039 respondents).
We anticipated chatter on Twitter and Hacker News, so we rolled the experiment out in small batches, keeping our eyes on the public noise.
The experiment involved 15 days of subject enrollment and three layers of content to measure (screener, user activity, and exit survey data), which made re-assembling the data complex, fun, and ugly!
Understanding attitudinal data
Our research team set out to measure three things:
- Coupon redemption
- Repository creation
- Perception of value
The first two (coupon redemption and repository creation) are what we think of as behaviors, whereas the third “perception of value” is less of a behavior, rather it’s attitudinal data. Attitudinal data conditions people’s reaction to the first two and overall promotion.
Both coupon redemption and repository creation are trackable behaviors with data that comes directly from the GitHub.com database: who clicked on our link, submitted the screener, created a new repository or migrated an existing one, and various other discrete activities.
“Perception of value” is what we refer to as attitudinal data, and comes from the exit survey. We designed the exit survey instrument to shine a light onto why people did or didn’t redeem their coupon and/or did or didn’t create their repository. By the time we are analyzing attitudinal data, we also had more information on respondent attributes (member of a free/paid organization, geography, OS, etc.).
Attitudinal data gives us greater insight into what levers we can pull on with marketing and product experiences to effect change in behaviors.
Understanding value
To our surprise, just over 4,000 of the possible 40,000 people redeemed their Golden Ticket and far fewer went the extra step to create repositories. We learned that people who were members of an organization on GitHub were more likely than their peers to take advantage of the offer. This told us something really important about how responsive different user types are to our outreach efforts, and how they use and value our product.
Insights:
- Value. Organization membership likely impacts redemption rates through attentiveness to notifications and awareness of GitHub’s value.
- Attention/Engagement. Members of organizations, particularly paid organizations, have relationships to communities that demand frequent attention and engagement. These members are likely to be more attentive to notifications from GitHub, which normally indicate project activity.
- Project work. Members of organizations may also be more likely to perceive value in private repositories, and have projects to work on that could benefit from a private space.
Our ability to influence this belief is conditioned by brand trust and loyalty, users’ own propensity for skepticism, and their use patterns- how they use Github, for what purposes, and who they use it with. Those are things we can’t measure with instrumentation in the app, so we have to rely upon self-reported data from surveys and customer interviews.
Qualitative insights
We were really excited about this experiment and really frustrated when it came time to wrangle and clean the data. The role of a researcher is to sift through many insights, but ultimately she should be able to find the story. We used answers from the category of “other” and the final open-text response option to help guide our analysis. Specifically, the very last question on our exit survey asked:
If we could grant you one wish to improve your experience with GitHub, what would you wish for?
Answers included:
at least five private repositories, or free private repos for teams up to five people
Free private repos for small teams, a la Bitbucket.
Private Repositories, for teams, for free
We realized that people were reading “free private repositories” as “private only to you” vs. “private to you and friends you invite to work on a code base.” (When we realized this, I personally smacked my forehead!) Would more people have redeemed their Golden Tickets if they knew they could invite collaborators to their private repositories?
Placing a value on GitHub Goods
People tell us that they want lots of things when they’re free, especially repositories. Asking respondents in this sample to choose between many valued things provides information on the ordering or prioritization of these preferences.
We asked both people who had redeemed but not used and those who hadn’t redeemed their free repositories, to share which of the most frequently requested goods would be most valuable.
Response options provided are deliberately disparate; not just things that we can or would actually provide (e.g. merged PR), but things that people find valuable on and around GitHub. Surprisingly, there’s not that much difference in these answers between the segments that redeemed their coupon and those who did not.
Our favorite open text response for the category of “Other” was easily:
Github underpants.
If you were studying value, what would you put on your product’s list?
Humans
Importantly, when we’re dealing with money we’re also dealing with humans. We confirmed that people don’t do things just because you send them a “free” product offer.
Humans do things because you send them something of value and they believe that using the thing you sent that will cause them to change their behavior will also benefit their lives. We additionally learned that delivering something by email, especially with the words “free” and “private” might not have gotten us off to a great start (note: in spite of sounding like salacious spam we verified that our deliverability was quite high).
Our ability to influence people’s beliefs is conditioned by the trust they place in the GitHub brand, their loyalty to the product, users’ own propensity for skepticism, and their use patterns–how they use GitHub, for what purposes, and who they use GitHub with. All of those are things we can’t measure with instrumentation in the app, so we have to rely upon self-reported data from surveys and customer interviews.
Key takeaways
- Feature awareness of the campaign and how to use GitHub is strikingly low. Many participants shared that they wished for “tutorials” to learn to use the free private repositories, indicating that “instructions are needed.” We also received feedback that summer vacations may have been a factor for why people missed the email or did not participate in the campaign.
- “Private repositories” may be an ambiguous description. A number of participants stated that they wish GitHub would provide private repositories for “teams,” so they could work with other people. We suspect that adding collaborators to private repositories is too opaque, and that “private” suggests working alone.
- One free repository feels special. While the number of repositories we offered (1, 3, or 5) to users did not affect redemption behavior, it did have a significant effect on use of the repositories. Those receiving just one free repository were more likely to use it than those who received three or five (25% versus 16%). However, in open-text comments we received feedback that one wasn’t enough for people who develop with modules, freelancers who have multiple client projects, and people asked for repositories that could be used by small teams.
—
Research is better when you get to discover new things about the world with someone who has a different perspective — someone who is willing to challenge you.
