Reinforcement Learning: Artificial Intelligence in Game Playing

In my previous post on recent developments in artificial intelligence, I identified reinforcement learning as one of the key and most disruptive machine learning domains.

The reinforcement learning is hardest part of machine learning. The most important results in deep learning such as image classification so far were obtained by supervised learning or unsupervised learning. Where in supervised learning, you can directly modify model based on target output, in reinforced learning, the target output is not available. You do not know, if turning left in a maze is good action immediately. You can however tell it after series of other decisions when an exit from maze was found. This is called delayed reward and it makes reinforcement learning so difficult.

You can use popular machine learning models (ensembles of convolutional nets, autoencoders, recurrent neural nets) in reinforcement learning but training the controller is much harder than in supervised learning world. Typically, you have to perform complex and time consuming simulations to get score of your model. This makes reinforcement learning tasks extremely time consuming.

Approximating the the Q function by a neural network

The core of most reinforcement learning approaches is learning a Q function that tell us utility of each state in the game. We can easily decide which action to take by choosing the action that lead to a state with higher utility (or fitness). Learning the Q function is hard. In fact, artificial intelligence researches are trying to find better way of representing and fitting the Q function for several decades. And the idea of using deep networks (both feed forward or recurrent) to model the utility function — this is what the deep reinforcement learning is about.

The game is changing now because sufficient computational resources are available finally. The research in the reinforcement learning field is gaining huge momentum mostly thanks to Google Deepmind. Some of the recently published results are impressive.

Game playing from visual input using deep reinforcement learning

Actually I was attracted to the machine learning field 20 years ago. I was looking for an innovative and self-adaptive controller for real-time strategy game.

At that time, it was too early, but now, the time has finally come and we have resources available to build truly intelligent opponent.

Deepmind prepared an environment for artificial intelligence to master realtime strategy Starcraft II.

There are many more challenges such as recent Visual Doom IA competition.

If you are not into playing from visual input, you can look at Angry Birds AI challenge, where guys from our lab scored very well multiple times. Another interesting environment was that for Ant AI challenge, where simple minmax based agents dominated due to very limited computational resources. Look what can be done by simple neuroevolution with indirect encoding:

Jan’s more recent paper in this field is here and they plan cool demo at NIPS 2016. Look at interesting papers accepted to the last DRL NIPS workshop.

Probably the most popular environment for training your reinforcement learning artificial intelligence is the OpenAI Gym.

You can learn how to build your simple artificial intelligence by reinforcement learning using the materials from the RL workshop available on github.

If you are beginner, look at awesome reinforcement learning tutorial by Arthur Juliani.

In case you are into general game playing look at this. The artificial general intelligence is even more interesting and ambitious so you might consider participating in this challenge.

We also open new course in February: