Self-Learning Data Analytics as a UX Designer: #3 Let’s do the most fun part! Summary a thing.

In the last chapter, I learned that before we achieve great results, we really need to be clear on what we would like to measure and try to craft our hypothesis very well.
P.S. If you missed the 1st and 2nd chapters, explore them here!

Chapter 1:

Self-learning about Data Analytics as a UX Designer: #1 Don’t Underestimate the Art of UX

Chapter 2:

Self-Learning Data Analytics as a UX Designer: #2 What to Do When Starting Tracking from Scratch

Then it becomes the part that I really looking for :)

After we have already set up analytics tracking on Google Analytics, it is time to start analyzing to address the points to which we want to find answers.

Let’s recap our hypotheses:

1. Which entrance is the most attractive to reach Ramakien?
2. There are 2 times for the introduction and onboarding but are both of them still engaging?
3. Is the Ramakien game too challenging?

#1 Which entrance is the most attractive to reach Ramakien?

Hypothesis: Based on students lacking an email to log in, the non-log-in entrance has a higher number of interactions than the log-in entrance.

Unfortunately, we couldn’t complete this analysis because we weren’t tracking events at the moment when people clicked on the card. I have brought the results from the traffic page, but I found that we can’t rely solely on results from page views and overall engagement because users can click on anything on this page, not just the Ramakien minigame.

By the way, there are still some assumptions based on the traffic page and qualitative data that we have collected.

Fact:

Point 1: Users engage more with the Minigame landing page compared to the main landing page. Although the main landing page has more views, the Minigame landing page performs better in terms of engagement metrics. Specifically:

• The Minigame landing page has more views per user (The Main Landing page = 6.49 vs the Minigame Landing page = 6.82).
• The Minigame landing page has a higher average engagement time (The Main Landing page = 41 seconds vs the Minigame Landing page = 59 seconds).

Point 2: There is a significant barrier for users to reach the Minigame Dashboard.

• Indicating that many users may not reach the Ramakien Minigame through this page. This barrier exists because accessing the Minigame Dashboard requires logging in, and many users, especially young children, may not have their own email accounts to log in.

In summary of Hypothesis 1

Assumption: The MiniGame Landing Page might be the place where most people access the Ramakien Minigame.

1. Based on the better engagement metrics of the Minigame landing page, it’s clear that users are more engaged with this page compared to others.
2. Additionally, there’s a low likelihood of users accessing the Minigame dashboard due to the requirement for an email to register a user account. Many users may not have an email address to fulfill this requirement.

#2 There are 2 times for the introduction and onboarding but are both of them still engaging?

Hypothesis:

Completing the introduction story and onboarding flow from end to end will indicate how engaging the introduction is.

We will know if this is true when:

1. Users spend time not less than 3 seconds per page due to reading.
2. We don’t see a high number of clicks on ‘Skip’ compared to the introduction step.

Fact:

There are two introduction stories, so I split the analysis for both of them.

Q: Is the introduction story 1 engaging?

1. The introduction story phase 1 is lengthy.

Most people only reach How to page no. 4(HowTo1_4), then 39% of users click ‘skip’ to the main game.

In other words, users drop out of the game once the tutorial begins teaching how to click the arrow.

Q: Is the introduction story phase 2 engaging?

Users show significant disinterest in the onboarding game, as indicated by a high rate of ‘skip’ actions after entering onboarding page 1.

• 70% of users from the total enter onboarding game phase 2 before starting to play level 7.
• 41% of users tend to skip to the main game after visiting the onboarding game’s first page [HowTo2_1].

In summary of Hypothesis 2

Assumption : The introduction story phases 1 and 2 fail to generate interest in learning.

1. Based on insights gained from schooling, where teachers typically instruct students first and students then teach each other to play, we can infer that the onboarding story may be unnecessary for user groups who play the game as part of their school curriculum.

#3 Is the Ramakien game too challenging?

Hypothesis:

Time spent on each stage and no. of interaction will indicate how difficult each stage is.

We will know if this is true when:

1. Users will spend less time in the earlier stages (1–6/10) of the MiniGame compared to the later stages (6–10/10) of the game. This is because, after level 6, users are required to perform more tasks, such as collecting items or crossing bridges, which necessitate increased interaction.
2. There is high interaction in the main game until users click the play result button.
3. There is a high number of users who reach the failed attempts page compared to other stages.

Point to validation 1

Users will spend less time in the earlier stages (1–6/10) of the MiniGame compared to the later stages (6–10/10) of the game. This is because, after level 6, users are required to perform more tasks, such as collecting items or crossing bridges, which necessitate increased interaction.

• Unfortunately, we didn’t track the amount of time spent, but we did analyze how many people win in the game compared to when they enter the stage level.

Fact:

Basically, adding more interaction after level 6 results in only a 12% decrease in the win rate.

• The average win rate for levels 1–6 is 71%, while the average win rate for levels 7–10 is 60%.

Insight 1: In games where users perform the same tasks but with longer paths, users will naturally experience an increased learning curve.

• Even though early levels may seem easy initially, as the game requires users to walk longer paths with each level, we observed that users achieve a high win rate at level 4 (81%) and level 6 (75%).
• This could be attributed to users learning from previous levels and adapting well to the challenges presented in levels 4 and 6.

Insight 2: Once we introduce the task of collecting items, which adds more complexity to the game, this step significantly increases the failure rate. Also, considering that users often lack interest in learning how to play the game through the introduction, it makes sense.

• We observed that at level 7, the win rate drops to 48%, whereas the benchmark win rate in this game is typically 68%.

Point to validation 2

There is high interaction in the main game until users click the play result button.

• Unfortunately, we didn’t track the amount of interection, but we did analyze how many people failed attempts in the game compared to when they entered the stage level.

Point to validation 3

There is a high number of users who reach the failed attempts page compared to other stages.

Fact:

The game is not overly challenging, as the number of failed attempts is usually consistent across levels, except for level 7, which experiences a notably higher number of failed attempts.

• The benchmark for failed attempts is 32%, but Level 7 has a failure rate of 52%.
• Even though levels 8–10 already require users to collect more items, the rate of failed attempts is almost the same as in the early levels (1–6).

In summary of Hypothesis 3

Assumption : The Ramakien game is generally not too challenging, except for level 7, where most users fail due to their lack of interest in going through the introduction.

1. Adding more interaction after level 6 only results in a 12% decrease in the win rate.
2. Levels 8–10 require users to collect more items, yet the rate of failed attempts remains similar to the early levels (1–6).
3. In games where users perform the same tasks but encounter longer paths, they naturally experience an increased learning curve. Although early levels may seem easy at first, users achieve high win rates at level 4 (81%) and level 6 (75%)
4. Once we introduce the task of collecting items, which adds more complexity to the game, this step significantly increases the failure rate, particularly due to users’ lack of interest in learning through the game introduction.

#4 Even though the game is challenging, is it still engaging?

Hypothesis:

Long time spent and high no. of interaction but users still keep continuing with the games, this will indicate how engaging the game is.

We will know if this is true when:

1. users spend a long time on stage
2. high no. of interaction.

But they still keep the tab ‘Next’ to the next level.

Unfortunately, I didn’t track both time spent and high interaction. However, I do have an event count for when people fail and lose all hearts (Mini Ramakien Game Failed 0) until they start to play the game again at level 1 until they win (Mini_Ramakien-Game_6_Reset).

Fact:

If a game is too challenging, it won’t have a significant impact, but it will likely encourage people to drop out after failing and losing all their hearts.

• There is no significant difference between the medium game level and the easiest game level in terms of users starting to play the game again after failing and losing all hearts. Typically, 80–90% of people who fail start the game again at level 1.
• However, at the hardest level, the percentage of people who start the game again drops to 60%.

By the way, people who try to play the game again after failing and restarting usually have a similar win rate, regardless of whether they come from an easy level or a tough level.

• The win rate from the easy level is 75% while the win rate from the hardest level is 77%.

In summary of Hypothesis 4

Assumption : If the game is too challenging in the case of the Ramakien game, people are still engaged, but it depends on which level they failed from. However, it won’t have a significant impact like losing all users once they fail in the game.

• At the hardest level, the percentage of people who start the game again drops to 60%, while people who come from mid and easy levels are typically willing to start the game again at 80–90%.

Suggestions from all learning

It seems that all evidence indicates a need to focus on the part that causes the most drop-offs for users, such as finding ways to make them interested in the introduction story by level 6.

• There might be some ways in which we can continue to do a quick proof of concept further. Ie. Rather than make an optional way to walk through the introduction before the game, we make it mandatory.

Lesson & Learn

1. In fact, while we work, we might not have much time to analyze and dig deeper into every point, but it depends on how accurate you need the insights to be.
2. Even if you can’t have all the data ready on day one, or sometimes it may not be entirely accurate, you can explore the context behind the scenes by conducting qualitative research. In my view, you really can’t rely solely on analyzing data or qualitative insights to guide product direction.
3. There is no perfect way to learn data analysis from day one; it may be a bit messy and challenging. However, after completing this project, I will have a better idea of what to focus on and the challenges I have overcome. :)

Big thanks to all my supporters

Thank you for your encouragement, guidance, and valuable suggestions:

Pete Chemsripong,Pee Tankulrat

Thanks for the valuable opportunity on playground data sources and your collaboration effort:

S.Songklod IToon,Tann Hiranyawech