Analytics without numbers: Viewing games through users’ eyes

And how this approach catapulted King of Thieves from disaster to global hit with 50 million players

Editor’s note: Oleg Yakubenkov is CEO of product management simulator GoPractice, has worked on some of the mobile gaming industry’s biggest hits, and spent some time as a Product Analyst at Facebook. In other words, he knows what he’s talking about.

Here are two things you’ll hear a lot from product teams:

1. A lot of data is needed to reduce uncertainty and get an accurate picture of users’ needs and behavior.

2. People working on products understand and know their users well.

The above statements are misconceptions, and in both cases, the reverse is true. In this post, I will discuss how analyzing user behavior without big data (debunking the first premise) will help us avoid the unpleasant outcomes of thinking you already know your users (debunking the second premise). I’ll support this with a case study of a game I worked on, King of Thieves, which, after a disastrous first launch, turned out to be a very successful game thanks to this approach.

Misconception #1: We need a lot of data to reduce total uncertainty

Imagine an infinity size pool of black and white balls. Your task is to find out the share of white balls. Initially, you are in a state of complete uncertainty. How much data do you need to find the answer to this question?

If you take 100 random balls and calculate the proportion of white balls, then you will know the answer with an accuracy of ± 9.8%. If you add another 100 balls, you will increase the accuracy of your answer to ± 6.9%; Add 200 more balls, you are up to ± 4.9%. Another 600 balls, up to ± 3.1%. With 9,000 more balls, you will achieve ± 1% accuracy.

Did you notice that the first 100 balls gave us a lot more knowledge than the next 100, and going from 100 to 200 balls had much more impact on the accuracy than the 1,000–10,000 range?

Let’s keep this in mind and move on to the second misconception.

Misconception #2: Game product managers understand and know their users well

The second misconception assumes we understand the products we are working on.

The truth is, we don’t.

We are always looking for ways to close the gap between the product model we envision and the actual product we build. To do so, we try out new tools and approaches, communicate with users, conduct experiments, and analyze data.

The goal behind all of these activities is simple. We want to improve our understanding of the product and users, to find opportunities for growth and development, and to increase the proportion of the right decisions we make.

Viewing the product through users’ eyes

What was this long introduction about? I want to introduce you to a method that will allow you to learn a lot about your product. It is very simple:

Take a few users and observe how they use your product.

No, you don’t have to conduct an interview with these users. There’s also no need to calculate the metrics that characterize their behavior in the product. We just need to take a specific user and monitor how they interact with the product throughout their entire period of usage.

How can you do it? If you already have event logging set right in the product, it is very simple:

1. Select the group of users you want to study (you will need different users to get answers to different questions).
2. For each user of this group, create a sequence of all the events they generate, including the event parameters, from the time they start using the product.
3. You will end up with a sequence of events for each user in your product. You can divide this sequence into separate sessions for convenience.

You can also do it with the help of Amplitude’s User Look-Up tool.

Now you have everything you need to view the game through the user’s eyes. Look at the sequence of events and start analyzing it. Take notes along the way.

You will observe what the user does while going through the onboarding and afterward, what difficulties they encounter along the way, at what point their first session ends, whether they have experienced the product’s value by that time, when they’re retained and how long it takes to do so, and much more.

Now, let’s go back to the first misconception. Studying a small number of users (50–100) with this method will help you find out the following:

1. How and why do people play your game?
2. What problems do they face at different stages?
3. How does the actual usage differ from what you intended?
4. Which users tend to stay and which tend to leave? What are the differences between them?

Another benefit of this approach is that it helps you come up with a bunch of hypotheses that you can test and validate through data analysis, experiments, or user research.

Pros and cons of analyzing user sessions manually

Analyzing user sessions will give you a more holistic understanding of how people are playing your game. This is not an abstract knowledge of how various popular features and conversion of various funnel steps work, but a vivid idea about your users’ paths within the game from the moment they start playing it until they either churn or it becomes a part of their life.

Another bonus of such an analysis is making it easier for you to work on new things in the product. Having a clearer picture of the current paths of your users inside the game, you can better predict how and at what stage the planned changes will affect them. This increases your chances of making the right decision.

Unlike using metrics, this method loses much less information along the way. But, as is usually the case, it also has its disadvantages. Unfortunately, you won’t be able to compare different versions of the product based on the manual session analysis method. You will have to analyze too many of them to obtain statistically significant results. In this case, metrics, cohort analysis, and A/B tests will help you.

You might also like: How a game I made in a weekend amassed 2M downloads using ASO

Case study: Finding problems in the first versions of King of Thieves

When creating products, we usually assume people are going to use them in the “correct” way. We design a product in a way that helps users discover its core value as soon as possible.

Surprisingly, “silly” users always do everything in their own way instead of the way we anticipate. This is exactly what happened with the first versions of King of Thieves.

King of Thieves is a mobile game in which you have to break into dungeons and steal treasures and gems. You can play the game in single-player mode, but the true value of the game is in its competitive multiplayer mode. The key feature of the game is to steal gems from other players to become the wealthiest thief. At the same time, players must defend their own treasures and build traps in order to stop rivals from stealing their treasures.

King of Thieves turned out to be a very successful game — but the launch of its first version was a disaster.

The game’s retention was a poor 26%. Most players left the game shortly after they tried it and never came back.

To understand the reasons behind this, we looked at the user sessions in the way described above. We wanted to understand what was wrong.

The core value of the game was in its multiplayer mode. When creating the game’s onboarding, we assumed that we would show how the game mechanics work in the first levels of the single-player campaign, and then players would switch to the multiplayer mode when they were ready. That’s exactly why we made a huge red multiplayer button after all!

But it turned out that most users simply went through the single-player levels without ever paying attention to the multiplayer game mode. After some time, they got stuck in a difficult level, or just got bored, and abandoned the game.

There were few who switched to multiplayer, but in most cases the players saw no difference from the single-player mode. The opponent either had no gems to steal, or they did, but when players completed the level, they weren’t awarded the gem, even though stealing gems from other players was part of the game’s core loop.

After analyzing the sessions of several hundred users (the sessions were very short, so it didn’t take us too long), we figured out that less than 10% of users had a chance to experience the essence of the game in their first few sessions. Those who grasped it were much more likely to keep playing the game.

Of course, we should not forget that at that point, this was just a correlation — it did not necessarily mean that players kept playing the game because of that experience. Nonetheless, it was a good starting point, and the hypothesis that walking more users through this experience will improve the metrics was worth testing.

We could have probably found this problem through metrics alone: We would see that a very small percentage of new users try the multiplayer mode. And in most cases, those who do try it do not steal gems from their opponents. But we didn’t know exactly what we were looking for or when we needed to pay attention. Perhaps everything was fine, and players simply didn’t find the mechanics of the game entertaining enough.

We launched a new version of the game in which we walked players through the multiplayer mode in the onboarding and also made sure that when players attacked a dungeon, there would be a gem for them to retrieve after completing the level. As a result, the game metrics significantly improved: Day 1 retention soared from 26% to 41%, while Day 7 retention increased from 9% to a healthier 20%.

While examining the sessions and looking at the product through the eyes of our users, we found the stark contradiction between what players experienced and what we wanted them to actually experience.

Summing it up

Y Combinator co-founder Paul Graham wrote an iconic essay titled “Do things that don’t scale.” Graham argues that founders should personally find their first customers and help them onboard into their product, because it offers the opportunity to look at your product through users’ eyes, which in turn allows you to quickly find points your users don’t understand, where they find the maximum value, and what is and isn’t important to them. Session analysis in mobile games (or any app for that matter) is a variation of this personal sales approach.

Many think that analytics is all about numbers and mathematics. In my opinion, analytics should help you to understand your product and answer questions you have about it. Numbers and metrics can help with this and solve some problems rather well, but you should not limit yourself to them. For some tasks, other approaches, including manual session analysis, will do a much better job.

Try examining the session of 50 users of your product. If you don’t find anything interesting, you’ve only lost a few hours of your time. However, and I can bet on it, you will probably find a few pleasant surprises.