How to Create Smarter NPCs in Games

Introduction

When it comes to video games, immersion is key. All types of games aim to keep the player immersed in some form or another. In role-playing games, it is important to create a believable game world in order to keep players immersed. A game world is made up of many things, but one aspect that is central to immersion is the other characters that live in the game world. These characters that fill a game world can be player-controlled or non-player controlled. This paper will focus on non-player characters (NPCs), how they contribute to immersion, how they work, and how they can be improved.

Background

NPCs and Their Role in Games and Immersion

As mentioned before, a non-player character (NPC) is a character that is not directly controlled by a player in a game. These characters make up part of the game world in games. A game world in a game is essentially the setting of the game. All role-playing games (RPGs) have a setting — such as the wild west, the medieval times, the distant future, etc. The NPCs in any given setting need to fit in to the world and behave believably based on the setting of the game in order for the game to immerse the player.

There are some common roles and tasks that NPCs fulfill in all RPGs. NPCs in games will offer quests, talk, play games, fight, and generally go about their business — seemingly of their own free will. They do all these things to create a game world that feels alive with real people in it, which is generally how they relate to immersion. When the NPCs in a game make the player feel like a believable facsimile of a person within the setting of the game, then the game has achieved immersion in regards to NPCs.

Immersion is quite nebulous and subjective, but is arguably the most important factor when measuring the quality of a game. Despite the difficulty in measuring immersion, Jennett et. al. presented findings that suggest there may be methods to measure immersion both qualitatively and quantitatively [8]. A measurement for immersion could be a valuable tool when assessing the quality of NPC artificial intelligence.

How Current NPCs Work

NPCs are typically controlled with state machines, which are a very common method of controlling various software and hardware. In video games, state machines are used to determine the behavior of AI agents. A state machine, at its core, is a directed graph. In an NPC state machine graph, nodes denote behavior states and edges denote events. A simple state machine example in a game is shown in Figure 1, which shows the state of an NPC guard in a fictional game setting. This state machine consists of two nodes, ‘Standing Guard’, and ‘Attacking’, and one edge, ‘Player Seen’. This graph is simply meant to illustrate what an NPC state machine could look like — typical NPC state machine in a real game would likely have tens, maybe hundreds of states. This form of behavior representation is fairly restrictive, allowing for only a few predetermined responses to events.

Figure 1: A simplified NPC state machine [9].

Another aspect of NPC behavior modeling in games is dialogue. Dialogue between player-controlled characters and NPCs is usually modeled by a form of a state machine called a dialogue tree. In a dialogue tree, nodes represent a response from an NPC and edges represent dialogue from the player. An example of a dialogue tree is shown in Figure 2. This model for controlling dialogue in games is fully dependent on game designers writing and animating every possible line of dialogue and then creating a digital representation of the dialogue tree for the whole conversation.

Figure 2: An example diagram of a dialogue tree [3].

Problems with Current NPCs

Imagine a scenario where you are playing a game and interacting with a guard that is controlled by a state machine similar to the one in Figure 1. You try sneaking by the guard, but they see you, attack, and then you run away. You try a couple more times to sneak by the guard with similar results. After a few more tries, you realize right where the blind spot is in the guard’s patrol route, enabling you to easily sneak past them every time. While it is not totally impossible for this situation to happen in real life, it is highly implausible. The state machine controlling the guard in this situation restricts the AI agent to performing only a few simple tasks and nothing more.

This situation illustrates the first problem with current NPCs: predictability. Because NPCs have a finite, predetermined set of behaviors (states), it does not take long for human players to learn these behaviors and exploit the mechanisms controlling the NPCs. This predictability breaks the player’s immersion because interaction with NPCs is not believable. In the example above, the player must come to one of two conclusions; either the guard is incredibly incompetent at their job, or the guard is not a real and is simply represented by some machine, breaking the player’s immersion.

This set of conclusions leads to another problem with current NPCs that is related to the first, which is the unrealistic reactions of NPCs in current games. NPCs cannot react realistically to emergent situations because their behaviors are predetermined. Continuing with the example from above, a more realistic reaction from the guard to repeated attempts by the player to sneak by would be to take some sort of extra precaution in guarding the area, making it harder for people to slip by. Or, perhaps the guard decides to enlist their guard buddies in laying a trap for the next person to attempt to slip through their area. For a human, it is easy for to come up with endless alternative responses to an emergent behavior, but this is not so for current NPCs.

How NPCs can be Improved

Fuzzy State Machines

Similar to a regular state machine, a fuzzy state machine is made up of nodes that represent behaviors and edges that represent events. However, fuzzy state machines add a new wrinkle: trait checks. Figure 3 shows a fuzzy version of the state machine in Figure 1, in which a relation check is used to determine the behavior of the NPC. In this particular model, the relationship check determines what state to transition to based on whether the player has a positive, neutral, or negative relationship with the NPC. In general, any agent trait can be used to inform a transition from one state to another in a fuzzy state machine. A small amount of randomness can also be added to a trait check to reduce the predictability of NPC behavior.

Figure 3: A fuzzy state machine for NPC behavior [9].

Relationship Graphs

A relationship graph can be used to represent relationships between characters within a game. In a relationship graph, nodes represent characters in the game and edges represent a relationship (with the absence of an edge representing the two characters not knowing each other). An example based on this idea can be seen in Figure 4 from Gruenwoldt et. al. that shows a relationship graph from their game Neuromancer.

Figure 4: An example of a relationship graph between characters in a game [9].

One variant of a relationship graph that I particularly like is the directed, weighted relationship graph. In this variation, the direction of edges indicates that the character at the tail of the edge knows of the character at the head of the edge. This allows for situations like celebrity, where many people know of a celebrity, but the celebrity does not know all of their fans. The other aspect of this variation is that each edge is weighted. The weight of an edge signifies some characteristic — for instance, how much a character likes another. Information about things like who knows of who, who is friends with who, etc. can be useful for fuzzy state machine to determine transitions to other states.

Procedural Facial Animation

Figure 5: A set of facial meshes for to be used as blendshapes for 3D facial animation. The larger faces on the left show the default/resting facial mesh. The smaller faces on the right each show a different morph target.

If an NPC in a game is to appear have human emotion then it must have appropriate, believable facial animation. Animated movies and cutscenes in many games use custom facial animation for each scene, which are either created by an animation artist or through motion capture technology. In the former method of facial animation, blendshapes [1], also called morph targets, are usually the industry standard. In order to animate with blendshapes, several copies of a facial mesh are created and deformed into different morph targets, each depicting a different facial expression or phonetic sound for the mouth (used to model speech). See Figure 5 to observe a set of facial meshes that make up the morph targets for a face mesh. Varying degrees of facial expression can be configured easily by linearly interpolating between the vertices of the default face and a specific morph target. Morph targets can even be combined to express mixed emotion. Given a set of morph targets for a face, any single frame in an animation can be expressed as a list of percentages, with each percentage corresponding to the weight of a corresponding morph target.

Typically, each frame in an animation is created by an animator, by hand. However, it is possible to remove the need for human frame-by-frame control through the use of procedural animation. In conjunction with some set of factors to inform the animation like a state machine, a relationship graph, and/or a set of personality traits, blendshapes can be used to procedurally generate facial animation. Emotional status can be determined at each state in an NPC’s state machine based upon recent events, previous state, personality traits, and the relationship status between the given NPC and the characters they are interacting with. When used together, the combination of these systems can create a reasonable facsimile of realistic, appropriate human emotion.

Core Personality Characteristics

Human behavior typically differs based upon a person’s personality. For instance, extroverts typically seek attention and human interaction, while introverts do not. One person’s reaction to a particular event will not necessarily be the same as another person’s because they differ in experience and personality. In order to reflect this aspect of humanness in an NPC, a set of core personality traits can be generated.

Figure 6: A graphical description of the ‘Big Five’ personality traits [13].

One set of personality traits that should be considered are a set commonly referred to as the ‘Big Five’, which have been collectively iterated on by researchers since the inception of this concept in the late 1950s at the U.S. Air Force Personnel Laboratory [14]. The five factor model was introduced in 1990 [4] and was further refined by several groups of researchers [2, 6, 11]. The traits that make up the ‘Big Five’ are: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. Figure 6 displays a short description of each trait.

To translate these personality traits into something a game engine can understand, each trait can be represented as a percentage. In this model, 0% represents that a character does not display the trait at all and 100% represents that a character fully displays a trait. With this encoding, these numbers can be used to inform state changes in a fuzzy state machine model. Recall that fuzzy state machines use agent traits to inform state changes. By representing each NPC with different, randomized personality traits like the ‘Big Five’, a game can achieve the effect of a unique personality for each NPC.

An Alternate Machine Learning

Machine learning is a powerful tool that can be applied to almost any problem space, which includes, theoretically, smarter NPCs in games. The great thing about machine learning algorithms is that they can produce a desired effect without the need for a human to design the algorithm. The problem with using them to create a smarter NPC in games is that they need to be trained on as large of a dataset as possible in order to produce the desired result. Said dataset must include input data and a desired output for each datapoint. For the task of creating a smarter NPC, it is challenging to define what a datapoint in a training dataset may look like. On top of that, compiling a large enough dataset to produce the desired results could be extremely laborious.

One method for training an NPC machine learning algorithm could be to utilize a machine learning algorithm to learn how to properly react to a given event. For this method, the input data for each datapoint would be an event, along with all relevant information for the NPC model to react to the event. Assuming the NPC includes all the NPC improvements listed above, the relevant information would include the given NPC’s relationship graph and their personality traits, along with any information needed to encode the event and its actors. The output could be rated on a scale of 1 to 10, with 1 representing a totally unrealistic reaction and 10 representing a completely realistic reaction. These scores would need to be determined and catalogued by a human observer manually.

The approach above is, admittedly, quite naive. Given enough time, this approach would eventually produce the right results. However, the human grading requirement severely limits how fast the algorithm can learn to produce more correct results. This approach is just one attempt at training a smarter NPC machine learning algorithm, with innumerable alternate approaches that have yet to be thought of.

Conclusions

While the gaming industry has continued to innovate in many areas over the years, NPC artificial intelligence has not been a part of that innovation. Current NPCs are controlled by basic state machines, and have been since NPCs first appeared in games. There are many problems with this basic model, most notably being predictability and unrealistic reactions to emergent behaviors. I propose four methods to increase immersion in relation to NPCs: fuzzy state machines, relationship graphs, procedural facial animation, and core personality traits, along with an alternate approach that involves machine learning. I hope to see changes to NPCs in the near future, and maybe even take part in the process of improvement and innovation.

Bibliography

1. Chuang, E., and Bregler, C. 2002. Performance driven facial animation using blendshape interpolation. Tech. rep., Stanford University.
2. Costa, P.T., Jr., McCrae, R.R.; Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI) manual. Odessa, FL: Psychological Assessment Resources, 1992.
3. “Dialogue Tree.” Ultimate Pop Culture Wiki, ultimatepopculture.fandom.com/wiki/Dialogue_tree.
4. Digman, J.M., “Personality structure: Emergence of the five-factormodel,” Annual Review of Psychology, 41, 417–440, 1990.
5. GameSprout. “GTA V — RANDOM & FUNNY MOMENTS 41 (Stupid NPCs, RKO!).” YouTube, YouTube, 2016, www.youtube.com/watch?v=jFswBl5dgBY.
6. Goldberg, L.R., “The structure of phenotypic personality traits,” American Psychologist, 48, 26–34, 1993.
7. Hi5M0F0. “Quotes of the Skyrim Guard.” YouTube, YouTube, 2011, www.youtube.com/watch?v=PI6hGG8ZX4c.
8. Jennett, Charlene, et al. “Measuring and Defining the Experience of Immersion in Games.” International Journal of Human-Computer Studies, vol. 66, no. 9, 2008, pp. 641–661., doi:10.1016/j.ijhcs.2008.04.004.
9. L. Gruenwoldt and M. Katchabaw. Creating Reactive Non Player Character Artificial Intelligence in Modern Video Games. In Proceedings of the 2005 GameOn North America Conference (2005).
10. Norman, Warren T. “Toward an Adequate Taxonomy of Personality Attributes: Replicated Factor Structure in Peer Nomination Personality Ratings.” The Journal of Abnormal and Social Psychology, vol. 66, no. 6, June 1963, pp. 574–583. EBSCOhost, doi:10.1037/h0040291.
11. Russell, M.T., Karol, D.; 16PF Fifth Edition administrator’s manual.” Champaign, IL: Institute for Personality & Ability Testing, 1994.
12. S. Watson, W. Banzhaf, and A. Vardy. Automated design for playability in computer game agents. IEEE Transactions on Computational Intelligence and AI in Games (CIG), 2014.
13. Theil, Edwin Van. “What Are the Big Five Personality Test Traits? — Learn All about the Theory.” 123test, 2020, www.123test.com/big-five-personality-theory/.
14. Tupes, E.C., Christal, R.E.; “Recurrent Personality Factors Based on Trait Ratings,” Technical Report ASD-TR-61–97, Lackland Air Force Base, TX: Personnel Laboratory, Air Force Systems Command, 1961.
15. Watson, Scott, et al. “Automated Design for Playability in Computer Game Agents.” 2014 IEEE Conference on Computational Intelligence and Games, 2014, doi:10.1109/cig.2014.6932860.