A/B Test-and-Learn with Your Players to Drive Product Strategy

How we used customer feedback to work on the RIGHT features

Demoing Mind the Trap at Casual Connect USA 2016.

Our most important value at Dissonance Entertainment is treating the player as our highest priority.

The customer is always right and it’s their opinions that you should value the most — not your own.

That carries over into everything we do, from brainstorming product features to bug testing. As a result of this mindset, we take every opportunity we can, such as demoing at trade conventions, to showcase our latest playable build to the public to get feedback and further drive the subsequent development of our in-game player experiences.

A scrappy, test-and-learn approach therefore has many benefits.

Avoid feature creep.

Feature creep is the tendency for product or project requirements to increase during development beyond those originally foreseen. Sometimes it’s unavoidable. Other times, it’s driven by an unnecessary desire to add features that the developer believes is needed. Feature creep becomes a massive headache and waste of resources when it’s discovered too late that the feature wasn’t even needed or wanted in the first place.

Frequently demoing at conventions gives you the ample opportunity to test these features early on and see how players respond to them before committing more resources. Most of the time, testers give valuable feedback on features they would like to see more of. They may not align with what you thought or what your business goals are, but once again, these are your future customers, so seriously consider them.

When we working on Mind the Trap, we wanted to keep it a local multiplayer “couch” game to stay true to the intimacy of playing with friends in the same room. But with each exhibition and marketing campaign, we received an overwhelming response to add online play. The following comment was very common:

If I can’t play online, I’m not buying this game.

Even though implementing online play would add another two months—probably more, since we had no experience with online networking—to development, it was clear that this was a huge selling point for our target audience, so we stayed true to our values and fit the feature into our project timeline.

Stay on track with the project timeline.

“Preparing for launch” is a pretty far out goal. For most indie console games, we’re talking 2-5 years of development time, and it’s pretty easy to get demotivated and lose direction in your work if there’s no immediate deadline to target for.

When consumers play a demo at a convention, there’s a reasonable expectation that the game looks like a game and is mostly playable. That means you have to polish up your demo, take care of game-breaking bugs, add in some music and sound effects, and incorporate art assets. Having a convention to demo at every months provides checkpoints for you to monitor your current progress, determine what the next plan is for your game, and maintain a more manageable plate before the big issues pile up.

Generate demand through early marketing.

Building a list of potential customers, such as email subscribers and social media followers, before launch is extremely important. These are the people who will follow your game’s progress, spread their excitement to their friends and followers, and be the first to purchase your game when it releases. Without these people, your game will be left in limbo until some famous Youtuber picks it up.

Here’s an example from Hitbox Games of what a sales figure graph looks like with a strong launch:


Create a vertical slice.

A vertical slice is a highly polished cross-section of your game to showcase what the final build of the game would look like. For most game companies, this is the demo, tutorial level, or marketing build.

Controversial? Yes, as we’ve seen with No Man’s Sky, but it’s still a resourceful strategy. Having a marketable build means you can create a stockpile of creative assets from which to use on trailers, giveaways, weekly social media content calendar, advertisements and website — all while the game is still in development.

Using a vertical slice also gives you a fully functional and marketable build to test with the public. Players don’t want to play a gray-box prototype — it’s ugly and looks like garbage. See the difference below.

Left: gray-box level. Right: vertical slice of same level

At a convention especially, the vertical slice attracts the attention of publishers and investors walking by, as it did for us at Casual Connect. The publishers that spoke to us weren’t necessarily impressed with the graphics; they were able to see what our final game would look like, determine if it was a good fit for their portfolio, and initiate a conversation on a potential partnership.

Lastly, having a vertical slice feels great because after weeks or months of doodling concept art and gray-box testing you and your team finally get to see your hard work come to life.

Conduct an A/B test and track analytics.

During our demo at the 2016 Sacramento Indie Arcade Gaming Expo, we coded our build to randomly turn new features on and off with each play session. The test group (sees new features) and control group (does not see new features) were compared against each other based on gameplay performance to measure the impact of each new feature on the gaming experience. In addition to monitoring the players’ emotional responses while they were playing, we used a custom-built analytics tool to track quantitative in-game variables, such as time spent per puzzle and distribution of points awarded, and were able to confidently determine which features would move forward and which would not.

Custom-built tool to track back-end analytics
Tracking the players’ emotional response to the new features at the 2016 Sacramento Indie Arcade Gaming Expo.