Reinforcement Learning: Artificial Intelligence in Game Playing
In my previous post on recent developments in artificial intelligence, I identified reinforcement learning as one of the key and most disruptive machine learning domains.
The reinforcement learning is hardest part of machine learning. The most important results in deep learning such as image classification so far were obtained by supervised learning or unsupervised learning. Where in supervised learning, you can directly modify model based on target output, in reinforced learning, the target output is not available. You do not know, if turning left in a maze is good action immediately. You can however tell it after series of other decisions when an exit from maze was found. This is called delayed reward and it makes reinforcement learning so difficult.
You can use popular machine learning models (ensembles of convolutional nets, autoencoders, recurrent neural nets) in reinforcement learning but training the controller is much harder than in supervised learning world. Typically, you have to perform complex and time consuming simulations to get score of your model. This makes reinforcement learning tasks extremely time consuming.
The core of most reinforcement learning approaches is learning a Q function that tell us utility of each state in the game. We can easily decide which action to take by choosing the action that lead to a state with higher utility (or fitness). Learning the Q function is hard. In fact, artificial intelligence researches are trying to find better way of representing and fitting the Q function for several decades. And the idea of using deep networks (both feed forward or recurrent) to model the utility function — this is what the deep reinforcement learning is about.
The game is changing now because sufficient computational resources are available finally. The research in the reinforcement learning field is gaining huge momentum mostly thanks to Google Deepmind. Some of the recently published results are impressive.
Actually I was attracted to the machine learning field 20 years ago. I was looking for an innovative and self-adaptive controller for real-time strategy game.
At that time, it was too early, but now, the time has finally come and we have resources available to build truly intelligent opponent.
Deepmind prepared an environment for artificial intelligence to master realtime strategy Starcraft II.
There are many more challenges such as recent Visual Doom IA competition.
If you are not into playing from visual input, you can look at Angry Birds AI challenge, where guys from our lab scored very well multiple times. Another interesting environment was that for Ant AI challenge, where simple minmax based agents dominated due to very limited computational resources. Look what can be done by simple neuroevolution with indirect encoding:
Probably the most popular environment for training your reinforcement learning artificial intelligence is the OpenAI Gym.
See ceobillionaire’s evaluation on LunarLander-v1gym.openai.com
You can learn how to build your simple artificial intelligence by reinforcement learning using the materials from the RL workshop available on github.
ML_workshop — Materials and organization for the upcoming ML workshop.github.com
If you are beginner, look at awesome reinforcement learning tutorial by Arthur Juliani.
For this tutorial in my Reinforcement Learning series, we are going to be exploring a family of RL algorithms called Q…medium.com
We also open new course in February: