The Challenge of Machine Learning in videogames

A thought that often crawls the mind of someone interested in videogame development is why the hell machine learning is not yet a standard in the godamn industry?
Artificial inteligence is often critized when the enemy stands, slowly dying, while you machiavellously snipe him from a distance so long your crosshair doesn’t even recognize him as an enemy. Most of the time, AI implementations of enemies are full of exploits that the player can take advantage of.
It may seem only natural that the solution to such problem is machine learning. Just imagine, after exploting the distance glitch described above on a few enemies, you attempt it once again without much though. Suddenly, you notice that the enemy acts differently. After trying it against a few other enemies, you can swear that enemies seem to have learnt how to hide from your cheese tactic. You find yourself even more surprised when one of them actually seems to be searching for you while hiding closer and closer.
Unfortunately, things aren’t that simple. There are several problems when trying to add learning into videogame AI. Kenneth O. Stanley, Bobby D. Bryant, and Risto Miikkulainen explain why in their article Evolving Neural Network Agents in the NERO Video Game.
First, there is the problem that comes with stochastic behaviour. With machine lerning, the AI needs to experiment sometimes new actions it can take to discover if what it normally does is, in fact, the best action it can make. The problem here: the AI actually needs to try new actions. This means that the AI could try to do some movements so random at random times that it may seem even more retard than a normal exploitable AI. Just picture the enemy AI stunt you with a flash granade, but instead of shooting you dead, he just decides to make a 360° turn and throw a granade to where his fellows are taking cover.
A solution to the stochastic behaviour is to train the AI before the actual release of the game. The problem with this solution is that the AI will not learn anything new at release and any new exploit found will remain there forever. Pre-training nullifies flexibility.
Real Time learning comes as a the most viable solution. The computer learning as the player plays would effectively solve any given exploit eventually. There are two problems here, however. First, machine learning is very expensive and Real Time Learning could have serious impact on the performance of the game, and nobody wants to play a slideshow of a game. Second, common machine learning techniques as back-propagation in Neural Networks learn really slow, for the entirety of the player’s playtime with a game, the AI could never show any type of learning.
This problems take us to the two crucial elements of machine learning in videogames: it must not impact performance, and it should learn really fast.
Techniques to achieve this milestones are still very experimental, and are not yet fully usable in mainstream game genres. However, the discoveries made so far are encouraging. Refering back to Stanley, Bryant and Miikkulainen’s work, we can learn about how they used NeuroEvolution of Augmenting topologies (NEAT) to create a game where you must train some robots for military tactics.
NEAT is a strategy to create Networks that can mutate. They start as simple neural networks (with no hidden layers) and mutate (adding new perceptrons for new potential features). If such a mutation proves no relevance the added neurone is deleted, if not, it can keep evolving. The learning can occur inbetween epochs (read play sessions) which minimizes the impact on performance. This solution also works fast because of the mutability (adding a whole new feature can impact far more than plain weight adjustment).
Again, NEAT is not perfect and cannot be applied to mainstream action first person shooters, but the ideas behind it encourage us to believe that one day we will see that utopic AI enemy overpowering us.
For more information about NEAT and it’s application, follow the next link: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.728.5120&rep=rep1&type=pdf#page=182