Machine learning in game development

AI got it’s game on!

Dota 2, the online battle arena video game by Valve, is amongst the most popular and challenging, cut-throat video games ever, wherein you have to coordinate and focus as a team of five. What’s interesting to look at is the AI connection it shares.

• In 2017, OpenAI with it’s project OpenAI Five defeated the world’s best players at the 1v1 mini game.
• In 2018, it defeated popular casters at a 5v5 in front of a live audience and 100k livestream viewers.
• In 2019, OpenAI Five won back-to-back games versus Dota 2 world champions OG at Finals, becoming the first AI to beat the world champions in an esports game.

According to a recent report by Newzoo senior market analyst Tom Wijman, the global gaming market will generate $159.3 billion in revenue in 2020 which surpasses all the leading industries. Looking at this humongous industry closely, we find that at the core, there are two aspects of a Game:

• Game Development
• Player Experience

1) Game development

Over the years, AI has mastered playing certain games. Try beating your computer at chess on the hardest difficulty, It’s pretty much impossible. Or even if you’re a pro StarCraft player, DeepMind software can crush you now.

But the AI inside of a video game has been building off of the same core set of principles for decades. Take, for instance, a classic game like Pac-Man. At different points, the ghosts evaluate where you are in the map and where you might be going, and then they either chase you, or they run away from you. It’s not exactly groundbreaking AI, but it is video game AI nonetheless. More so, what’s remarkable is that the AI you encounter in the games today has remained unchanged over the course of time. Two of the core components of commercial game AI are pathfinding and finite state machines.

Pathfinding is how to get from point A to point B in a simple way, and it’s used in all games all the time.

Finite state machine is a construct where a non-player character can be in different states and moves between them.

a) More Realistic Interactions & graphics

In the past decade, game developers have revolutionized the games by delivering realistic experiences. There was a time when we played games in 2D, many games now use 3D rendering, which results in vastly superior visuals, dynamic simulations and even VR/AR, to change the way games are played today completely. Game developers today have tools that have made it possible for them to create visually appealing, interactive games. A team from the Computer Science and Artificial Intelligence Lab at Massachusetts Institute of Technology and NVIDIA recently demonstrated how it is possible to generate synthetic 3D gaming environments using a neural network that has been trained on real videos of cityscapes.

This process, called video-to-video synthesis, involves getting the Machine Learning (ML) model to learn how to best translate input source video into video output that looks as photo-realistic as the original video content.

To achieve this, the team based their approach on previous work like Pix2Pix, an open-source image-to-image translation tool that uses neural networks. In addition, the researchers utilized a particular type of unsupervised deep learning algorithm called Generative Adversarial Networks (GANs), which designates one neural network as a “generator” and another neural network as a “discriminator.” These two networks play a zero-sum game — with the generator network aiming to produce a synthesized video that the discriminator network cannot ultimately determine as fake

Machine learning also made it easy to convert a low resolution game model to a high resolution by adding more pixels.

b) Smarter NPCs

Making a game smarter means getting intelligent reactions from the non-playable characters (NPCs) inside the game. It means that making the bots play in a human way.

For example, the way humans drive cars is very different from the way a bot would. So, to simulate that, we add variables which replicate human behavior.

There are many algorithms like The Division’s AI-driven path-finding, City Conquest’s playtesting via genetic algorithm and more, to make games smarter. While some developers use scripting to control NPCs, algorithms and AI systems like above assist in managing these. What’s unique about using AI and algorithms is that these NPCs also learn from the actions taken by the player and change their behavior accordingly. Leveraging AI and ML capabilities, game developers in the future can create games that can develop better NPCs and manage their behaviours within the game using AI and ML algorithms like Steering Behaviors, Finite State Machines, etc.

c) Well balanced game

Machine learning techniques have been widely used in competitive domains, to find a strategy which maximizes the payoffs for the agent in most scenarios of the competition.

For example, we have to make a bot for a game of Valorant. We add three variables and assign values to them, which together provide the information of the current game scenario to it. Let’s say, the map is assigned to variable 1, the character is assigned to variable 2 and the possible actions the opposing character can perform is assigned to variable 3. So, the output will be based on a machine learning algorithm which decides the action to be performed by the bot according to the skill level of the opponent with least possible error.

Computer games can be seen as highly competitive environments, however, in this case, it is necessary to achieve a balanced behavior. Game balancing is related to ensuring a good level of challenge in a game, which implies avoiding the extremes of getting the player frustrated because the game is too hard or becoming bored because the game is too easy. The goal is to keep the game level adapted to the performance of the human player, no matter his or her skill level, which can vary widely from novices to experts. Unfortunately, fixing a few predefined and static difficulty levels (e.g., beginner, intermediate and advanced) is not sufficient. In fact, maintaining the adequate level is a dynamic process, because of the evolution of the players’ behavior, as a natural consequence of the experience acquired in playing the game. On the other hand, as user skills can regress (for instance, after a long period of not playing the game), regressions of the level are also needed.

For this Q-Learning, a popular Reinforcement Learning (RL) algorithm is mostly used to address dynamic game balancing. This is divided into two separate problems: Learning (building agents that can learn optimal strategies) and Adapting (providing action-selection mechanisms for inducing game balance). Due to the immediate interaction with humans, including experts, offline training is needed to bootstrap the learning process. Then, online learning is used to adapt to human opponents, in order to discover the optimal strategy to play against him or her.

Standard Q-Learning, when not doing exploration, selects in the action-selection mechanism, the action whose value is maximal for the current state. In fact, the agent chooses the best action for each situation and keeps learning in order to improve its performance. In our case, we cannot simply keep the agent acting as best as possible.

For a given situation, if the game level is too hard, the agent does not choose the optimal action (the one with highest value, as given by the action value function constructed in Q-Learning), but chooses progressively sub-optimal actions until its performance is as good as the player’s. This entails choosing the second best action, the third one, and so on, until it reaches the player’s level. Similarly, if the game level becomes too easy, it will choose actions whose values are higher, possibly until it reaches the optimal one.

2) Player experience

a) In-game Communication

The gaming audience has rocketed in number while simultaneously expanding globally. In today’s time, players are able to connect with each other from all across the world. But people aren’t always as friendly to each other on the internet. And all this does is to create an unpleasant experience for other players. When you’ve got one person dominating the conversation, or a group of people that are being hurtful to others, it can cause serious problems.

Types of anti-social or disruptive behavior (often referred to as “toxic” within the gaming community) include “griefing”, chat spamming, bug exploitation, and cyberbullying (including racial or minority harassment).

The way developers treat this sort of scenario these days is by providing a mechanism for players to report other players, report their bad behavior or any kind of misconduct during the game. At this stage, you gather a bunch of diagnostic information, maybe some chat logs, maybe their in-game recordings, and so on, and you pass it off to a team that has to triage it. That’s a manual effort. Triaging that sort of work takes a lot of time. For filtering inappropriate conversations, they use LSTM and Bi-directional LSTM (BLSTM) sequential models. The proposed models do not rely on hand-crafted features, hence are trained end-end as a single model. Google made an API called the Cloud Natural Language API, which can actually detect sentiments in individuals’ chat messages. So, as a way of swiftly triaging through reports, you can quickly identify where you may have some problem areas in your logs, and make that triage process a lot simpler.

b) No explicit content

Lots of game developers are providing ways for players to create user generated content. It might be items, custom images and even maps. Turns out that if you give people the ability to upload whatever they like, they can, again, upload things that are probably not appropriate for everyone there. Flaws in the automatic adult video detection is a problem of interest to many organizations around the world. The aim is to restrict the easy access of underage youngsters to such potentially harmful material. Most of the existing techniques are mere extensions of image categorization approaches. Developers have proposed a video genre classification technique tuned specifically for adult content detection by considering cinematographic principles. Spatial and temporal simple features are used with machine learning algorithms to perform the classification into two classes: adult and non-offensive video material. Game developers have achieved an accuracy of 94.44% by using different SVM classifiers. Google developed an API called the Vision API and it is able to do things like object detection, and is also able to flag explicit content.

c) Fraud Detection

While artificial intelligence has entered the mainstream in a big way and found applications in almost every walk of life, there is one field where it has been widely used for many years. This is in the online multiplayer video game space, where algorithms have been widely used since the early 2000s. This is used by cheaters to give themselves an unfair advantage over human players, ruining the atmosphere of the game and the way it’s meant to be played. Cheating in online games is a problem both on the esport stage and in the gaming community. When a player cheats, the competitors do not compete on the same terms anymore and this becomes a major problem when high price pools are involved. To tackle this, video game publishers have been trying to fight the rising tide of online hackers. However, the rise of the Internet has resulted in the ability for cheats to be spread widely, incentivising developers to take a heavier hand towards moderating cheaters in online communities.

While there are numerous exploits that can be used in video games, there is one that is used prominently and employs machine learning in its advanced stages. These algorithms are known as aimbots, and they rely on the data player’s client computer receives regarding the other players on the server. Following this, the program then finds whether they are visible from the player’s position, and if they are, uses a program known as the Triggerbot to fire a bullet. The more advanced versions of these algorithms use image recognition to detect whether a player is in the view of the other player or not. This model is trained on multiple images of the textures used in the game to enable it to differentiate between say a wall and a player model. Some aimbots also utilize ML for the dynamic learning of movements exhibited by the opponent and for learning previously unknown maps. These kinds of systems fall under the category of an AI expert system that tries to mimic how an expert human would play a game.

For example, a bot was created in the game Counter Strike: Global Offensive to detect hackers. The machine learning approach was taken by creating a recurrent neural network with data from games played with the cheat aimbot and without the cheat aimbot. Basically, humans were pretty good at determining when a player was cheating or not, even though the cheat couldn’t be identified by VAC. Valve decided that with this data, they could use deep learning in the form of neural nets to build a program that could detect cheats in the same way a human would. VACnet, as it was later named, used the data from the investigator convictions from Overwatch to train the model, and constantly ran this model across 3500 processors to scan the 150,000 daily matches played on Valve’s CS:GO servers. Early results showed that conviction rates increased from 15–30% to 80–95% (close to 100% when newly re-trained), but VACnet results were ultimately given to a human to determine guilt and the appropriate punishment.

In Conclusion, machine learning and deep learning have acted as a catalyst in Game Development. In the near-future, we’re going to see game directors that learn to adapt the game as you are playing it, and learn to become game masters that play the player as the player plays the game.