A/B Test-and-Learn with Your Players to Drive Product Strategy

How we used customer feedback to work on the RIGHT features

Kenneth Ng
Aug 22, 2017 · 5 min read
Demoing Mind the Trap at Casual Connect USA 2016.

Our most important value at Dissonance Entertainment is treating the player as our highest priority.

The customer is always right and it’s their opinions that you should value the most — not your own.

That carries over into everything we do, from brainstorming product features to bug testing. As a result of this mindset, we take every opportunity we can, such as demoing at trade conventions, to showcase our latest playable build to the public to get feedback and further drive the subsequent development of our in-game player experiences.

A scrappy, test-and-learn approach therefore has many benefits.

Avoid feature creep.

Frequently demoing at conventions gives you the ample opportunity to test these features early on and see how players respond to them before committing more resources. Most of the time, testers give valuable feedback on features they would like to see more of. They may not align with what you thought or what your business goals are, but once again, these are your future customers, so seriously consider them.

When we working on Mind the Trap, we wanted to keep it a local multiplayer “couch” game to stay true to the intimacy of playing with friends in the same room. But with each exhibition and marketing campaign, we received an overwhelming response to add online play. The following comment was very common:

If I can’t play online, I’m not buying this game.

Even though implementing online play would add another two months—probably more, since we had no experience with online networking—to development, it was clear that this was a huge selling point for our target audience, so we stayed true to our values and fit the feature into our project timeline.

Stay on track with the project timeline.

When consumers play a demo at a convention, there’s a reasonable expectation that the game looks like a game and is mostly playable. That means you have to polish up your demo, take care of game-breaking bugs, add in some music and sound effects, and incorporate art assets. Having a convention to demo at every months provides checkpoints for you to monitor your current progress, determine what the next plan is for your game, and maintain a more manageable plate before the big issues pile up.

Generate demand through early marketing.

Here’s an example from Hitbox Games of what a sales figure graph looks like with a strong launch:


Execution

Create a vertical slice.

Controversial? Yes, as we’ve seen with No Man’s Sky, but it’s still a resourceful strategy. Having a marketable build means you can create a stockpile of creative assets from which to use on trailers, giveaways, weekly social media content calendar, advertisements and website — all while the game is still in development.

Using a vertical slice also gives you a fully functional and marketable build to test with the public. Players don’t want to play a gray-box prototype — it’s ugly and looks like garbage. See the difference below.

Left: gray-box level. Right: vertical slice of same level

At a convention especially, the vertical slice attracts the attention of publishers and investors walking by, as it did for us at Casual Connect. The publishers that spoke to us weren’t necessarily impressed with the graphics; they were able to see what our final game would look like, determine if it was a good fit for their portfolio, and initiate a conversation on a potential partnership.

Lastly, having a vertical slice feels great because after weeks or months of doodling concept art and gray-box testing you and your team finally get to see your hard work come to life.

Conduct an A/B test and track analytics.

Custom-built tool to track back-end analytics
Tracking the players’ emotional response to the new features at the 2016 Sacramento Indie Arcade Gaming Expo.

Kenneth L. Ng

Developer at Alibaba

)

Kenneth Ng

Written by

AR/VR Developer at Alibaba

Kenneth L. Ng

Developer at Alibaba

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade