Data-Centric Recommendations for Growing an Online Community for Developers
What was this project all about?
Our collaboration with DEV this semester was focused on how to grow the site, both from a business and community perspective. Conversations around conversion, retention, and personalization all centered around the core goal of site growth. While we’ve all used social media and online publishing platforms, it was our first time tackling what turned out to be a hard question: How do we define site growth, and what levers can DEV pull to impact said growth? This blog post will hardly do justice to our entire analytical journey justice, but by-and-large, our analysis involved exploring different engines for site growth, and defining and analyzing proxy metrics.
All aboard the User Flywheel!
Figuring out where in the user flywheel to begin our analysis was a challenging first-step. Afterall, if writers beget readers, who beget engagement, which begets more writers, at what stage do you jump in to begin your analysis? There was no clear single answer. We ultimately elected to begin by defining an active user, as our first proxy-metric from site growth.
How do we define active users?
Our goal with defining DEV’s active users was to inform our understanding of retention, which would also help us understand and explain user churn (why, when, and where users leave). The first step was to conduct some qualitative analysis to learn about how similar companies define their north star metric (i.e., 1 metric most predictive of a company’s long-term success). We found that many similar businesses aligned with DEV’s model focus on leveraging monthly active users (MAU). From there, we explored user actions across time to identify factors aligned with site growth, with the finding that site growth is defined by increased page views. To confirm this finding, we conducted a correlation matrix among Month_new_pageview and user activities (e.g. comments, publishing etc). The strongest correlation found is between number of page views and number of articles (i.e., 0.84). Thus, an active user is defined as: a user who writes at least one article per month, and we deemed article writing the highest level of engagement for a user in the DEV community.
If an active user on DEV is an article writer, how do we keep article writers writing?
Now that we defined an active user based on article writing, we wanted to know how well DEV retains article users. Our analysis found that monthly retention for active users drops below 20% one month after writing the first article and 10% three months after writing the first article, across all recent cohorts. Knowing that retention of article writers can be improved, we created a sankey diagram to understand the different levels of engagement and movement between viewers / writers / churned users. The split between writers:viewers:churnedusers is 20:50:30 at month 1 and 10:40:50 at month 3. Re-attracting users (viewer → writer, churned user → viewer) is unlikely, so it is best not to lose active users in the first place.
Finally, a logistic regression was performed to ascertain what motivates article writing to ultimately understand how to keep users active. We found that follows motivated users (i.e., regression showed explanatory power of follows on future article writing). The proposed solution was to open up follower count and increase transparency of followers. To date, DEV has wanted to hold off on introducing potentially-competitive platform dynamics, so has not implemented that suggestion yet.
Note: We were also able to apply a similar analysis to commenters, another important role in the DEV community. Like article writers, commenters tended to become less active over time. We ran a logistic regression and proportional z-test, and found that commenters with at least one positive reaction on their first comment were 30% more likely to comment a second time within 15 days. Additionally, the commenters that are more likely to comment more than once typically will comment within 3 days of their most recent comment. A potential action DEV can take is to incentivize positive reactions, or to send notifications encouraging article readers to react positively to article comments.
Should DEV users bother with tagging their articles?
Another area we delved into was the impact on tagging. DEV currently allows up to 4 tags per article, but doesn’t require users to input tags. When conducting exploratory analysis comparing popular and unpopular articles on DEV, we noticed that popular articles almost always had tags, while many unpopular articles did not. Additionally, approximately ~15% of all articles on DEV did not have tags. Tugging this thread further, we ran a t-test on two groups — articles that had 1+ tags associated, and articles with 0 tags. Our results are below:
The results of our t-test show that on average, articles with tags perform better than articles without tags in all major engagement metrics. Most notably, articles with tags on average garner >200 more views than articles with no tags. We concluded that based on this analysis,tagging articles effectively increases article performance. From an intuitive community engagement standpoint, the positive impact of tagging makes sense, as tags allow DEV users to find and engage with the articles they are interested in (e.g. beginner coder searching for articles tagged for beginners). Our next steps for DEV was to recommend some form of a UI change that would prompt users to add at least 1 tag before posting an article, or make tagging compulsory. An example mock-up is below:
Given that this suggestion requires a relatively low amount of tech effort to change, DEV will be implementing a UI change into the site to prompt users to include tags in published articles. We expect this change to decrease the proportion of un-tagged articles, and potentially increase the overall average view count per article published on DEV.
A Warm Welcome — Our analysis of engagement within DEV welcome threads
Consider this thought experiment: when a user first signs onto Tinder, the infamous and original swipe-to-like dating app, their profile is immediately shown to as many users near them as possible — in hopes of increasing their chances at receiving an elusive “match”, ASAP. The concept behind this was simple: the sooner a user gets a match, the sooner they’ll experience the accompanying thrill and know what Tinder is about.
While there is no swiping on this social media platform, there are opportunities to interact with new users. For one, DEV was exploring expanding “Round Robin” notifications — notifications nudging certain existing users to interact with new users. Secondly, DEV hosts weekly Welcome Threads, where new users can introduce themselves to the community. What we endeavored to prove was: Does a new user receiving positive interactions in the welcome thread mean they will have more engaging lifetimes on DEV? Said differently, can interacting with new users in this welcome thread achieve the same effect as an early “match” on Tinder, and keep the new user engaged on the platform?
While our preliminary analysis showed that new users who post in the Welcome Threads have more lifetime platform engagement than users who are absent from the Welcome Threads… that wasn’t very helpful. One can imagine that the Welcome Thread is a self-selecting population of eager new users who are always going to have more engaging lifetimes on DEV. Therefore, we further divided the Welcome Thread population into “hyped-up” users, who received a high number of replies and reactions to their posts, and “non-hyped-up” users, or users who received low levels of engagement on their posts.
A T-Test confirmed our hypothesis. “Hyped-up” new users in the Welcome Thread outperformed their peers who received fewer interactions on the Welcome Thread posts. Well after the Welcome Threads have expired and throughout their lives on DEV, these “hyped-up” users comment more, post more, visit more pages, and post more articles, than their “non-hyped-up” peers.
Ecstatic about our findings, we hope to see not only more new users introducing themselves in the weekly Welcome Threads, but also see existing users excitedly welcoming these new members into the DEV community — encouraging them to share their ideas and meet new people. After all, who doesn’t feel a little more empowered when a kind stranger says “Hello!”.
Thank you!
We hope you enjoyed reading through our data-analytics journey as much as we enjoyed living it. A special thank you to the DEV team for the opportunity and being a delight to work with, as well as our professors and TAs for their support and guidance throughout the semester.
