Measuring and Improving Social Health in Mobile Gaming Communities
Rachel Zhang, Julian Runge, Hernan Silberman, James Marr, Katherine Kiourellis, Mark Williams, Dan Barnes
“Man is by nature a social animal.” These words are thousands of years old and as true today as in the antique world when Aristotle perceived of them. Humans are naturally oriented towards others and seek social interactions. We live, work, eat, and entertain in groups. Arguably, all entertainment derives from (mediated) social interaction.
With ubiquitous adoption of smartphones, social interactions are massively facilitated. Growing virtual communities offer broader and more diverse opportunities for people to connect online, through various apps that serve as intermediaries, and at any time. Online games where a substantial number of people participate in the game simultaneously, are an essential venue for such communities.
In this post, we showcase an approach to support social health in Legendary: Game of Heroes. Our approach uses a concoction of institutional knowledge, social science, and machine learning techniques. On the social science-side of things, we resort to an assortative matching approach that matches players based on their behavioral similarity. Assortative matching has been popular in marriage and labor markets since the 70s — more on this below. On the machine learning-side, we leverage XGBoost for user-level predictions and a simple geometric mean-based classification to identify healthy social environments. Our ultimate goal is to build a fun-maximizing community for players coming from diverse backgrounds to bond frictionlessly and evolve and explore the game together.
1. Leveraging Institutional Expertise to Identify ‘Healthy’ Social Environments
Guilds are vital to players’ experience in Legendary. (Sidenote: Two players who met in a guild in Legendary, got married and thanked us by sending photos from their marriage.) Players join a guild pretty much right after they complete the game’s initial tutorial. Joining a guild allows players to bond with other players, to learn from each other, to exchange in-game gifts, and to collaborate in game-wide competitions and events.
With tens of thousands of active guilds, Legendary has an active and rapidly growing guild community. Each guild is led by a guild leader and has up to 30 members. In this project, we aim at finding the best guilds to recommend to players who newly join the game by defining a classifier to identify ‘healthy’ social environments that help players mutually enrich their game experience. Institutional knowledge plays an important role for this identification. Experienced product managers have game-specific domain expertise and a deep understanding of what constitutes an exciting guild environment. Prior knowledge given during conversations with institutional experts laid the foundation of our classification of guilds. We focused on aspects such as number of active players per day, number of missions played together, and the volume of gifts and messages exchanged between guild members (Figure 2).
We then combined various input features providing a holistic view on ‘health’ of a social environment into a ‘health score’ using a weighted geometric mean, and set a cut-off on that score to predict future healthy versus not-super-healthy environments. In essence, we built a very simple classification model of guild health, routed in institutional knowledge and readily interpretable by domain experts. The aim of the resulting guild health score and classification is to detect the most active and fun environment for new players to be placed in, to in turn maximize their enjoyment of the game.
2. A First Take at Improving Players’ Social Experience: Assortative Matching Theory
Guilds are composed of players, so the ‘health’ of the social environment is ultimately determined by the activeness and growth of its members. Besides classifying guilds into two general health categories, how can we match players to their “best” teams such that the bonding between players is the most stable and fun-maximizing?
Assortative matching theory proposes that players will be motivated to spend time with peers that share similar preferences, for example, similar commitment to the game so that they grow at the same pace to become stronger and benefit each other when fighting missions together. The concept was introduced by the Economist Gary Becker in the context of the marriage market in 1973 (Becker 1973). Under the assumption that each individual is doing their best (maximizing utility) and the marriage market is stable (in equilibrium), Becker found that “Men differing in physical capital, education…and many other traits will tend to marry women of LIKE VALUES of these traits.” After Becker, the concept of assortative matching was extended to studying household behaviors, firm formation, and the labor market, where partnering of agents occurs naturally.¹
In a first simple take, we decided to adopt a similar approach in our player-guild matching schema. Matching a player to a guild becomes, in essence, matching a player to a team of players that share similar goals, activeness, and dedication to the game.
On the guild side, we already constructed the guild health score classifying guild environments by differing degrees of ‘social health’, e.g. more active and demanding environments vs. more relaxed and easygoing environments. Now, to enable assortative matching, we need a similar model on the player side, predicting players’ future activeness and dedication to the game.
3. Using Machine Learning to Predict Player Types
The prediction of different play styles from users’ behavioral traces as commonly collected by app publishers is a supervised learning problem. We can use historic data to identify different play styles and then predict these play style labels from the behavioral traces that users generate right after app download, before reaching the guild recommender screen shown in Figure 1.
Among a number of different machine learning approaches, regression tree-based XGBoost surfaced as the best predictor. All models were trained on the same historic user data, with input features containing information collected at app download, and in-app user activities up to the point the user became eligible to join a guild. They were then evaluated on a time-independent hold-out dataset to compare performance.
The XGBoost model deployed to production continuously makes predictions of engagement and dedication values for new users (figure 4). It retrains itself daily on the newest training data available, and automatically performs feature selection and parameter tuning.
After obtaining a prediction for each new user entering our game, we map these prediction values to categories consistent with the available guild health environments. The thresholds of this classification are set to equilibrate supply and demand in each preference (in terms of activity and dedication) category. Thus the matching market is in equilibrium where demand (number of players looking for a guild spot) is equal to supply (number of available guild spots).
4. Applying the System in the Field: Increased Social Activity
Our declared goal was to increase players’ joint enjoyment of the game. So, did the system impact player behavior in online field runs in the desired way? We are happy to report that it did. Concretely, the number of chat messages sent and received by new players went up by 25%, as did other in-game and guild activity metrics.
Assortative matching hence seems to be effective in catalyzing players’ game and social experience. Our recommendation of the most suitable guild for players to join likely also minimizes the cognitive load and search friction that can be caused by a mismatch between team and player types. An assortative matching approach is further likely to provide the best social learning environment for players to understand the game in-depth. We hence expect positive impact on longer term engagement metrics and will keep monitoring.
5. Beyond “The Likes Attract”
While assortative matching is a great starting point, there may be more sophisticated matching approaches with potential to create even more enjoyable social environments. For example, assortative matching can run the risk of creating highly homogenous social groups. Speaking to this, heterogeneous matching has been a popular approach, e.g. for grouping students in classrooms. Numerous studies² in the education literature point out that low- and average-ability students can be motivated by learning from their high-ability peers and benefit significantly from heterogeneous groupings. This could hold true in an online gaming environment as well where less skilled and less engaged players may benefit from a team with a few highly skilled and engaged players to learn from. We hope to find out soon — stay tuned.
[1] Theoretical papers that study the assortative matching model under different conditions are e.g. Durlauf (2003), Besley and Ghatak (2005), Legros and Newman (2007), Chai et al. (2016) and Ahlin (2017).
Observationally, Luechinger et al (2006) studied matching of employees to firms and Choi et al. (2008) investigate matching of guids and players based on social- or task-oriented preferences.
Experimentally, assortative matching is studied in public goods games and prisoner’s dilemma: Page et al. (2005), Gunnthorsdottir et al (2007) and Rabanal and Paul (2012) are examples.
[2] Hooper and Hannafin (1988), Saleh, Lazonder and Jong (2005), Falk and Ichino (2006), Mas and Moretti (2009).
