Hey! I’m Cem and I’ve been developing games since 2018 — you can read a little more about me here! I started my company OnRush Studio in 2021 which has since grown to 10 team members! I worked with Poki in 2019 as a solo developer and then once OnRush was created, we started working closer together. We have developed many titles for the web gaming market, including some in genres not seen before on web such as Venge.io and Tribals.io (multiplayer survival). These days, we have 10 million gameplays monthly which was helped by our new data-driven approach to games which we started last year.
We started developing casual titles to learn more about this data-driven approach. Before, we were developing our games with gut feeling, experience, and player feedback. Don’t get me wrong this approach is great for game development, but OnRush has 10 people now and we wanted to focus more on the company’s growth. We needed a better approach, one not completely reliant on gut feeling or luck. Things needed to be deterministic.
Making some core changes in our approach started to show results already…
When we started developing our first casual game, Poki was working on an experimental tool called the ‘playtest’. I signed up for it and started to experiment with it.
In this post, I will share some insights from our playtest processes and how we use them with our internal data tools to analyze data. This tool provides us with video footage from a random player’s gameplay, which helps us to understand more about the player’s experience.
We are still learning! Some of this might be wrong, or there could be better ways of doing these things so please comment under the post and share your thoughts!
Playtest tool
This is what the playtest tool looks like. Each session is played by a random Poki player and recorded on Poki’s systems. We watch each video carefully and try to make sense of the players’ behaviors. Each video also has keyboard strokes, mouse movement recordings, and console logs with timestamps!
This tool helps us to understand what’s going on in the session without the user being interrupted. We can’t ask questions as it’s not live, the session is recorded and sent to the dashboard. Users are not forced to play our game, they play because they are interested in playing this game in the first place. The opt-in to the playtest only happens when the player is in-game which changes everything.
When players are forced to play a game, they often intentionally play longer. As this is a random test system, if they hate the game they just leave. This shows their true intentions and feelings towards the game so the data is much more robust. On the flip side, this is a disadvantage because we can’t understand why exactly they left, but there are ways to figure that out as well!
This is our sheet to track each version and take notes about player’s behavior. We aim to resolve a pain point in each iteration to increase the engagement time by fixing issues or making experiences better in the game. Sometimes it works, sometimes it doesn’t work. Making bold claims from data only is hard, which is why this process might take weeks and months. That’s why we iterate!
Iteration Framework
We try to address one metric per iteration. When we try to test multiple things, it’s hard to pinpoint what caused the drop or increase in metrics so we need to select what to iterate on each version carefully.
First of all, we only fix the bugs we encounter in the error collection system or if we see a weird issue in the playtests. We also fix UX issues and make interfaces and game controls easier for players. Thankfully, the playtest shows us everything about the player’s keyboard and mouse behaviors which helps a lot because UX issues can be hard to understand sometimes. In these cases, we have internal meetings and brainstorming sessions to come up with unique solutions.
If it’s a bug, then it must be fixed. If it’s a UX problem, it needs to be addressed. These are obvious ones. But what if we want to add a feature and test it on the next iteration? Features are hard to implement and it’s hard to tell which ones will be successful. We have a simple calculation to help us figure out our next move.
- Impact: How impactful this change might be? (0–10 score)
- Time: How much time it will take to develop? (0–10 score)
We put all of these tasks into our idea list repo with scores and start with the highest. But that’s not all! Before starting to work on a big feature, we do another test to see if it catches the attention of players and if they are interested in it. We do fake implementations to check the potential or show a popup to see if players would like the feature. We also conduct more playtests to collect data percentages about that specific feature.
Our data tool clearly shows the interest in certain features. Some features get the attention of players, and some of them are not interesting at all. At the end of the day, we are developers and players often have different perspectives about their experience. The curse of knowledge bias affects us in game development a lot!
Measuring the potential before starting to develop a certain feature is key to using time efficiently. After collecting data, we sort the list again and focus on the features that matter the most!
Non-obvious Becomes Obvious
When we play and test our games, we test every feature every time. It becomes a tedious, boring process but we find some bugs and issues which is normal when you play a game again and again — they’re part of the experience! However, because we are constantly playing our own game and are so close to it, we can miss some of the most annoying bugs in the game so you always need new eyes and new players to see the experience from 0.
The playtest feature provides you with new, random players who haven’t tested your game before. Thanks to this, even the bugs we ignored while developing the game can be seen in the videos super clearly. Every time we watch playtests, we say AHA! and fix it in the next iteration.
Some Insights from Playtests
- It helps you to out find what’s wrong with your product, anything that you can’t see clearly
- It helps you to come up and brainstorm better ideas with player’s intentions at the forefront
- It shows interesting bugs and glitches that players encounter that could be hard to reproduce
- It shows a fresh gameplay experience each time you run a test, this way you can perfect tutorials and first levels easily
- It allows you to test different variations, so you can find the best possible option for the player experience or monetization
- Any many other things we are still learning today!
Also, player experiences vary. Every play style is different, even in the simplest of games. Seeing different play styles helps you design better experiences for your game.
In our last game, we noticed that players were leaving the game suddenly while playing. The playtest feature shows the video stopping suddenly. We were a little bit upset about the results but we noticed that this pattern could be seen on every video so we went back to the drawing board and did a brainstorm. We came up with some ideas, but none of them were obvious fixes. One of the implementations was locking the mouse so that when players shoot, they can aim at the object they are shooting easily. Releasing that feature bumped the engagement from 2 minutes to 10 minutes almost overnight! At this point, the non-obvious became obvious. Players were trying to shoot but with the excitement and stress of the game, their mouse was going outside of the game area and they were clicking another game thumbnail unintentionally.
When I tell this story it seems obvious, but at that moment it wasn’t. We didn’t even think about implementing such a feature in the first place. The playtest allows you to find interesting insights about your game which, as a developer in your own bubble, you might miss. Every time I watch a random player playing my game, I always think: Was that even possible?
Final
Playtests are helping us a lot to perfect our products. For indie studios like ourselves, it’s hard to build our own infrastructure but tools like Poki’s playtest opened a new window to our development and allowed us to better understand player behavior. Companies like Valve have been using this method for years. Here is a good video that shows their secret weapon, playtesting:
I hope you enjoyed this post! Please comment underneath and share your thoughts about our process and what we can do better!
You can follow me for more on Twitter: https://twitter.com/cemdemir
