Image for post
Image for post

How to Measure the Success of Your Game Design

Achievement rates are a powerful tool for tracking and analyzing player engagement

Josh Bycer
Oct 23 · 9 min read

I have recently been taking a closer look at the analytics that come with video games — more specifically, the achievement rates on games purchased via Steam. Looking at them has given me some insight into the hallmark of what truly makes a good game, and it has nothing to do with the number of reviews, accolades — or surprisingly — the number of sales. Today, I’m talking about player retention when it comes to single-player games, and what we can learn about how people continue to play (or quit) a game.

The Critical Contradiction

Before we begin, it’s important to mention that we’re not talking multiplayer or live service games in this article. Games that have further content added post-launch retain players differently than single-player experiences.

Developers and gamers alike are often quick to point out what they think is the metric for success with a game. Maybe a title had a huge opening week and sold millions of copies, maybe it won GOTY, or every YouTuber and streamer can’t stop talking about it. The problem with these metrics is that they don’t tell the story when it comes to something game developers need to know — did their game design work? Game development is a constantly iterative process; if you’re not growing as a designer, then you will fall behind. As I have said countless times, one success does not magically guarantee all future ones — such as the failed “Bioware magic” piece on Anthem. And before anyone says it, the number of copies sold does not necessarily correlate to game design success as I’ll demonstrate in this piece.

Image for post
Image for post
For Marvel’s Avengers, the drop-off rate is consistent, and almost a 40% completion rate of the main game. This means that while people had problems with the game, it did manage to retain the audience throughout the campaign.

There’s a reason why when we talk about game success it’s often hard to put into words why one game did well as opposed to another. That one sentence could easily frame multiple posts and books, and something I continue to try and answer. What we often see surrounding the best games is this sense of a full package: great gameplay, great aesthetic, and everything just kind of fits into place. When a game is good, it keeps people invested in playing it, akin to reading a good book or binge-watching an entire series. What I’ve learned and have seen is that just because people are talking about a game, or that it has a lot of reviews, doesn’t make that a great game. To study the quality of a game and how well it did, we need to talk about how the fans reacted to it.

Burning Fans

It’s time for a salient point that every developer needs to hear: Most people are not going to finish your game no matter how much praise it gets. This isn’t a slam against designers, but just the nature of the market and playing games. No matter what the game there will be drop-off when it comes to fan engagement. This can happen fast, it can happen slowly, but looking at the achievement list of games tells the tale.

People can stop playing a game for several reasons. Some people will load up a game, try it for a few minutes, see it’s not for them, and uninstall it. Maybe there’s a really hard section or massive difficulty spike that drives people away. For some, the game could be over when they’ve had their fill no matter how much more of the game is left to play. There is no one right way to make a game (just as there’s often no one right way to play them), but drop-off is often the sign that something didn’t click with players.

Image for post
Image for post
For Doom Eternal, the completion rate sits at 41.3 which is really good considering the polarizing opinions on the design and difficulty of the marauder fights. As another fact, only 2.6 of the playerbase on steam got the battle mode achievement for 25 matches, which proves that most of the players were not interested in the multiplayer mode.

Again, even the best games released each year will have drop-off to them. What I look for when scanning achievements is the percentage of people who finished the game, and when we say “finished,” we mean to the credits, not 100% clear. The other metric is the drop-off spikes — when does a major part of the playerbase stop playing and how much was it? I’ve seen games that have been considered critical successes have 80% of their playerbase stop playing within an hour. There are games where less than 5% of the people who played it saw the game through to the very end.

What we can learn from these stats is how successful the designer was at keeping people invested via their gameplay and story. Major drop-off points can represent difficulty spikes, pain points in the design, or just where the game faltered in terms of engagement. With that said, there are a few other aspects I want to discuss and what they mean for designers.

Calculating Consumers

Before I reveal this next statistic, here’s a question for readers: Without looking below, in terms of finishing a game, what percentage rate of consumers would you consider “good?”

From examining some of the most popular and not so popular games, I’ve settled on the following. If a game has a 50% or higher clear rate, that is fantastic. Great games tend to average 40–49%. Most games sit in the 30–39% range which is good, and anything less than that would be poor.

Roguelikes kind of skew these results to some extent because clearing one run is often just getting started with playing the game. However, there are plenty of roguelikes with low clear rates. Whereas Slay the Spire was around 75% of beating the game once, and then the drop-off occurring when it came to its ascension mode.

Image for post
Image for post
Now for Celeste, I find this one very interesting. Despite all the praise people gave it, and the inclusion of the assist mode, very few people actually finished the game. What we can take from these stats is that many players got their fill of the mechanics early on, and not even the assist mode was enough to motivate them to push through to the end.

Keep in mind that when we rate games in this capacity, the game’s reception is not factored in. Hollow Knight may be one of the best indie games of the last decade, but it has a poor clear rate sitting at 18% on steam. The question then: What can we say about Hollow Knight given its overall praise, but low clear rate? Again, the Steam achievements give us a good idea. Hollow Knight had some quality of life issues when it came to playing it and learning it. From the get-go, the game lost 26% of its fans before they even got the first charm. Conversely, Dark Souls 3 lost 18% of its fans at the first boss fight, and Hollow Knight lost a total of 32% of its fans for its first boss.

That tells us that there was enough of an issue with Hollow Knight’s design that a good portion of people couldn’t overlook it and progress further. The next burnout spike is a 10% drop going from the first boss to the second, and then another 17% drop to the third. Here’s the thing — those drops occurred for most players within about two hours of playing the game. Losing that much of the fanbase in that span of time is not good.

As I said, most people are not going to finish a game, but we also want to see if the burnout rate is a gradual drop instead of a sudden spike. Dark Souls 3 has one spike dropping from 82% to 63% at the game’s second boss. However, from that point forward, the drop off becomes a consistent amount — lowering 2–4% for the bosses going forward. Even the game’s difficulty spike with Pontiff Sulyvahn had a clear rate of 45%

Image for post
Image for post
Spelunky 2 may have gotten praise from a lot of sites and people, but the achievement page tells a different story in terms of making progress in the game. There has already been a balance patch, and I wonder if we’ll see more.

For every game, no matter what, the drop-off rate will stabilize for the fans who are “in it” over the long haul. This percent usually sits around 30% or lower for a lot of games. These are the fans that will try to do everything in your game before giving up. At the 10% or less metric, we have the super fans who will go for 100% clear. Studying this is where the challenge of knowing who to listen to about your game comes in. If the hardcore fans say that your game is perfect and not change a thing, but they only make up less than 10% of the people who bought it, you may need to rethink some of their suggestions.

Room to Grow

To wrap things up for this piece, there’s one final point I want to talk about when it comes to game development. There’s a tendency for people to become complacent with success in all walks of life. For games, we have seen developers fall into the echo chamber effect that their games are “perfect” or “the best game ever made” if they blow up. For games that become critical and commercial darlings like God of War, Death Stranding, Horizon Zero Dawn, Prey 2016, or The Last of Us Part II, critical discourse is often swept under the rug in favor of just outright praise. It’s easy to lump critical breakdowns of games with people going on mad tirades about it. Or a developer will say that the people who didn’t finish the game “just aren’t fans.”

Image for post
Image for post
Despite being an amazing game, very few of Hollow Knight’s fans saw it through to the end.

Unfortunately, not hearing about the issues is how game developers hurt themselves going forward.

There is always room to improve and grow when it comes to game design, no matter how great your previous game was. An amazing game keeps people invested while cutting out or mitigating as many pain points as possible. Right now, the game that has had the highest clear rate I’ve seen is Ori and the Will of the Wisps sitting at 52%.

Just listening to the praise and ignoring everything else will keep your hardcore fans happy, but not grow that base. I have not enjoyed any game from Bioware going back to Jade Empire, as I saw this echo chamber of not wanting to look at or improve playability concerns from someone who isn’t already a Bioware CRPG fan. Each game I played, the general gameplay just did not work for me. We often praise Blizzard in its heyday for listening to fans and critiquing their titles and designs. As I have said, there is a difference between someone not liking a genre and not liking your take on it.

Image for post
Image for post
Conversely with Hades, despite having a huge drop off early in the game, it does retain more people who managed to beat the final fight. With that said, Supergiant should look into why there was such a huge drop off before finishing the first area.

There are always ways to improve the design, and keeping your head in the sand while other people are innovating and growing is a surefire way to fall behind. This is part of what lead to the indie explosion last decade — indie devs making innovative and amazing titles that major studios did not want to risk money on. Listening to feedback and seeing where you can improve can go a long way towards making a better game. Remember, just because someone isn’t complaining, doesn’t mean that your game is perfect.

Image for post
Image for post

Finally, something for you to consider. The next time you finish a game on Steam, check out the achievements page and see how many people cleared it. For developers, pay attention to the drop off spikes and try to remember if those points were noticeable for you.

If you enjoyed my post, consider joining the Game-Wisdom discord channel. It’s open to everyone.

Image for post
Image for post
The original version of this article appeared on Game-Wisdom. It has been revised and published at SUPERJUMP with permission.

SUPERJUMP

Celebrating video games and their creators

Josh Bycer

Written by

Josh Bycer is the owner of Game-Wisdom and specializes in examining the art and science of games. He has over seven years of experience discussing game design.

SUPERJUMP

SUPERJUMP

Celebrating video games and their creators

Josh Bycer

Written by

Josh Bycer is the owner of Game-Wisdom and specializes in examining the art and science of games. He has over seven years of experience discussing game design.

SUPERJUMP

SUPERJUMP

Celebrating video games and their creators

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store