Measuring Successful Game Design (part 2)

I am too lazy to go get another image…

Note: This is the second and final part to “Measuring Successful Game Design.” If you haven’t already, check out part one here.

In the first part of “Measuring Successful Game Design,” I discussed why understanding whether or not you’ve been successful as a designer can be challenging. I also talked about putting on a researcher hat when observing play tests, and how to get your money’s worth when doing so. In part two, I’ll be focusing on surveys as a method to better understand what is or isn’t working with your design, with particular attention paid to the survey I use when play testing the game I’m working on, Diabolical! At the end of the article, I will also provide a copy of the survey for your personal or professional use.

Surveying play testers

While observing play tests is the best way to understand what is or isn’t working with your game, it has one key limitation. Observing a play test cannot directly tell you what a player’s perceptions of your game are. Though some players may tell you their thoughts without needing to be asked, for the most part players will not offer criticism naturally. And those who do offer criticism may not know how to do so in a way that is constructive to what you’re trying to make.

There are two real ways to get at this information: interviews and surveys. While interviews have a lot of benefits — namely the ability to ask why a respondent feels a certain way — I have opted to use primarily surveys both in the interest of time for my play testers as well as my own time. Interviews are a much more time-intensive way of gathering data, as you have to meet and converse with each respondent. Also, since interviews tend to be less structured than surveys, you frequently will want to take the feedback through qualitative coding analysis to get more structured, analyzable data.

While surveys do have some limitations, they are a great way of better understanding what’s going on in someone’s head as they interact with a system. In order to get the most out of it, though, you must understand how to write effective survey questions. There are a million guides out there to writing good questions for a survey, so if you’re interested in crafting your own survey I recommend taking a look at those. Here, I will be going over the survey I use, and will be discussing in some depth why I chose to put on there the questions I did. When referencing a specific question on the survey, the question itself will be in bold, and the available answers will be in [brackets].

The survey I use when play testing Diabolical! is made up of 5 general sections:

1. Basic information about the player’s familiarity and experience playing the game, as well as their general board game playing habits
2. A modified version of the System Usability Scale (SUS)
3. Questions relating to the player’s perceptions of their choices while playing the game
4. Free-form feedback
5. Basic 1—10 rating of the game overall

I will go over each section as well as the questions in it to describe the survey in more detail.

Basic information about the player’s familiarity and experience playing the game, as well as their general board game playing habits

This section — which makes up 4 of the 10 questions in this survey — helps me understand the respondent a little bit better and gives me some information that I can use to interpret some of the other sections more effectively.

1. How many times have you play tested this game? [1–3 times; 4–6 times; 7–9 times; 10 or more times]

Knowing a player’s familiarity with what is being tested is important for a couple of reasons. First, people tend to like things more as they become more acquainted with them. This is known as the Familiarity Heuristic. As such, I can expect players’ responses to skew a little more towards the positive as they test the game more. This will also help me understand a bit more about the depth of the game. For example, if I were to slice the data to separate out the veteran players and find that they were reporting significantly more boredom while playing, I could interpret that to mean that — while the game may be fun for newer players — it may not have enough depth to keep people interested for longer.

2. How often do you play board games (including card games and tabletop games)? Exclude play testing this game. [Very rarely; Rarely; Every so often; Frequently; Very frequently]

This question helps me comprehend how much of a “gamer” this respondent is. This question helps me interpret the curb appeal the game has to those who don’t play games as often, and can also help me ensure I am keeping the rules simple and easy to understand. For example, let’s say I were to single out respondents who said they play board games “Rarely” or “Very Rarely.” When I do that, I find that they are rating the game as far more confusing than those who are more familiar with board games. Knowing this, I can take a number of actions to make the game less confusing to new players. For instance, I could provide cheat sheets for how a turn goes, or I could revisit the verbiage in the rulebook to make sure I’m not using language that relies on a player’s preexisting familiarity with terms used in other games.

3. Did you win the game? [Yes; No; We didn’t finish]

It can be much easier to have a good time with a game when you’re on the winning side of things. To me, the mark of a great game is one that you can still have a great time playing even if you don’t win. By knowing whether or not the respondent won, I can see if there are any significant response differences from those who win a game compared to those who lose it. I can also keep an eye out for any game-breaking strategies a player may use in future play tests or in their free-form responses.

4. How quickly did the game move during this play test? [The game went by very quickly; The game went by quickly; The game was neither fast nor slow; The game went by slowly; The game went by very slowly]

This question is specifically looking at a player’s perception of time while playing the game, not the actual time. I time the length of every play test, so I already know that. A person’s perception of time, however, can say a lot of interesting things. The adage “time flies when you’re having fun” is relevant here. If the game doesn’t take long to play, yet feels slow, then there’s likely to be a significant amount of down time or boring moments for players.

A modified version of the System Usability Scale (SUS)

The SUS is a quick, 10 question survey which pertains to a user’s perceptions of the usability of a system. It has been in constant use — mostly in software usability research — since 1986. The SUS gives respondents a series of statements and asks them to rate their level of agreement from 1 (Strongly Disagree) to 5 (Strongly Agree). Since it was designed for digital purposes, I modified some of the language so that it relates better to the context of playing board games (mostly by replacing instances of “use” with “play” and “system” with “game”). These questions give me insight into how confident players feel in their ability to understand and interact with the game.

5. Please indicate your level of agreement with the following statements. [Strongly Disagree; Disagree; Neutral; Agree; Strongly Agree]

• I think that I would like to play this game frequently
• I found the game to be unnecessarily complex
• I thought the game was easy to play
• I think that I would need the support of an experienced player to play the game
• I found the various elements of the game well integrated
• I thought the game was too disjointed
• I would imagine that most people would learn to play this game very quickly
• I found the game very cumbersome to play
• I felt very confident playing the game
• I needed to learn a lot of rules before I could get going with this game

I decided to use the SUS because I was aiming to create a game that was not super complex at face value. While some players enjoy extremely complex games with tome-like rule books (such as Warhammer 40k), the goal of Diabolical! is to be accessible to a wide variety of gamers. Diabolical! is intended to have a forgiving learning curve that can be enjoyed by hardcore gamers and casual gamers alike.

Questions relating to the player’s perceptions of their choices while playing the game

The element of choice is one of the foundational pieces of good game design. A good game seeks to present players with enough viable choices at any point in the game that they don’t feel stuck, while not presenting so many options that they feel overwhelmed. Furthermore, the choices a player makes must have clear enough feedback so that they can quickly understand whether their choices are bringing them closer to victory or further from it. Like the SUS, this section gives respondents 10 statements and asks them to rate their level of agreement.

6. Please indicate your level of agreement with the following statements. [Strongly Disagree; Disagree; Neutral; Agree; Strongly Agree]

• I thought my choices as a player affected the outcome of the game
• I felt overwhelmed with options
• I felt there were enough options from turn to turn
• I thought it was obvious what choice I had to make at any point in the game
• I felt like I could catch my opponents off guard
• I think it’s difficult for players to catch up if they are losing
• I felt I could adjust my strategy effectively
• I feel there are not enough winning strategies
• I felt I could easily anticipate my opponents’ moves
• I think this game is boring

Each of these statements has implications for how the element of choice is being portrayed in the game. By analyzing these statements, I can learn if a player had too many or too few options, if the “right” option was too obvious, and if the choices a player was making seemed to do anything useful.

Free-form feedback

While the survey to this point has been fairly extensive, it is by no means exhaustive of all the types of things a respondent may want to tell me about their experience playing the game. To make sure I had an avenue to collect any information I wasn’t gathering in the more structured portions of the survey, I included three free-form questions for respondents.

7. Was there anything about the game that stood out to you in a positive way?
8. Was there anything about the game that stood out to you in a negative way?
9. Do you have any additional comments or feedback that you’d like to provide at this time?

These questions allow players to more freely and openly express their thoughts. As I gather more data, I can analyze it in aggregate and look for recurring themes across responses. This data can give me valuable information about things I should be capturing but currently aren’t, or changes I should make to the game that I hadn’t considered.

Basic 1–10 rating of the game overall

10. On a scale of 1–10, how likely is it that you would recommend this game to a friend?

The final question wraps up the whole package of the players’ overall experiences into a single number. While this number may or may not have a strong correlation with the more specific feedback I’m gathering in other sections, it gives me a solid number that I can look back on and (hopefully) see grow over time.

Wrapping up

Surveys are a great way to understand how play testers are reacting to your game. By asking the right questions in the right way, you can gain a treasure trove of information to help drive your decisions as a designer.

However, even though surveys are a great way to get information, they shouldn’t be the only way you get that information. It’s best to use a mixture of different information gathering techniques as each has its own unique plusses and minuses. I mainly use direct observation and surveys, as well as interviews to a lesser extent.

By using research methods to effectively grasp the state of your current game, you will be far more able to address the game’s issues and create something that will be enjoyed by gamers everywhere!

Thanks for reading; I hope you enjoyed. If you’d like to download a copy of this survey for your personal or professional use, you can do so here. Special thanks to my friend and researcher Saul for helping to edit the survey and keeping my questions on point. If you enjoyed this post or found it valuable, please use the heart icon below to recommend!