Conquering OpenAI Retro Contest 1: Preparing Everything for the Contest

1 What is Open AI Retro Contest?

OpenAI Retro Contest is a just released Meta Reinforcement Learning contest. The goal of this contest is pretty simple while extremely difficult: that is to solve the Sonic Games of SEGA Genesis:

Captured from https://contest.openai.com/

Unlike conventional reinforcement learning setting (which is only trained and tested on the same game), test game levels are different from train game levels in the Retro Contest. Such setting makes the problem much more challenging, which means that we have to obtain some prior knowledge when learning on train levels and then utilize such knowledge to learn new test levels as fast as possible. This RL setting/task is so-called Meta Reinforcement Learning .

The goal of OpenAI to hosts such a cool contest is obvious, that is to raise people’s attention on Meta Reinforcement Learning. It is really a right time to do research on this new AI research field.

2 What is Meta Reinforcement Learning?

Captured from https://contest.openai.com/

I directly capture above picture from https://contest.openai.com/ to describe Meta Reinforcement Learning. Traditional reinforcement learning always face s a sample inefficient problem, therefore a DRL agent (such as DQN) has to train millions of time steps to solve ATARI games. However, we humans can solve the same game in just a few minutes. Why we can do that so quickly? I think the reason is obvious, that is we have prior knowledge about the game. We can easily come out with some basic playing strategies when we play a game at the first time, while current DRL agent only trains to play games from scratch. Therefore, a new research problem comes out:

How to make Deep Reinforcement Learning agent learn faster?

Deep Meta Reinforcement Learning is a potential solution!

By mimicking humans’ behaviors, we hope our neural network agent can also learn some prior knowledge from different tasks (training tasks), then utilizing such knowledge to learn new tasks (testing tasks) fast, shown as above pictures. This is just Meta Reinforcement Learning’s definition.

However, since this research problem is brand new, there is not a suitable benchmark to test it. Therefore, it is so cool that OpenAI made such a great benchmark by playing the Sonic games.

From the latest literature about Meta Reinforcement Learning from Deepmind: Prefrontal cortex as a meta-reinforcement learning system, we can find that our brain is somewhat a meta-reinforcement learning system, our phasic dopamine(DA) implements model-free RL while our prefrontal cortex (PFC) performs model-based RL. This finding is pretty glamorous, if we hope to achieve AGI, we must do research on Meta Reinforcement Learning. My enthusiasm of Meta Reinforcement Learning also comes from such hypothesis. I believe Meta Learning leads to AGI.

3 Preparing Everything for the Contest

So, here we go! Let’s conquer the Retro Contest! But at the very first, we have to set up all the necessary work environment for this contest.

OpenAI have written a basic tutorial for us: https://contest.openai.com/details However, there are some detail tricks that we have to deal with.

Here I summarize some essential steps to build the environment:

Step 1: Basic Requirements

Prepare a computer/server with GPUs.

System: Ubuntu 16.04

You must have installed basic deep learning required packages such like CUDA, cuDNN, python3, pip, git, tensorflow, pytorch etc.

Step 2: Install Docker (for training locally and uploading agent dockers to OpenAI contest server)

You much have a root priority so as to install Docker. I installed the latest version of Docker via .deb (downloaded from website, followed by website):

$ sudo dpkg -i /path/to/package.deb

Step 2: Register a Retro Contest Account to login in Docker

Once you registered, you can have such information in your profile:

Then you can login in Docker on your computer:

export DOCKER_REGISTRY="retrocontesteruhyafkwbdnjpoq.azurecr.io"
docker login $DOCKER_REGISTRY \
    --username "retrocontesteruhyafkwbdnjpoq" \
    --password "wMf1HBNeRgTmya3YWGyoIu+P7INdmc7t"

Step 3: Install Docker images

We have to install several docker images to run experiments locally and prepare to build docker images for uploading:

sudo docker pull openai/retro-agent
sudo docker pull openai/retro-agent:pytorch (if you use pytorch)
sudo docker pull openai/retro-env
sudo docker tag openai/retro-env remote-env

sudo docker pull nvidia/cuda

we must installed nvidia-docker if we want to use GPU in Docker. Follow this git to install it!

Step 4: Install all necessary gits.

You can just follow OpenAI’s tutorial to install retro-contest, retro.

Step 5: Buy the Sonic Games on Steam

Yes, we have to pay some money to buy all three games. So we have to register a Steam account and buy them. Then, we can use below script to load our games to Retro:

python -m retro.import.sega_classics

Step 6: Write our agent, build, test a Docker and Upload

We have installed all necessary packages. Then we can write our own agent (codes).

Step 6.1: prepare all your codes

Step 6.2: write a docker file. retro-baselines is a nice example for us to follow.

Step 6.3: build a Docker like this:

docker build -f agent.docker -t $DOCKER_REGISTRY/agent:v1 .

Step 6.4: test the Docker locally:

1) Without GPU:

retro-contest run --agent $DOCKER_REGISTRY/agent:v1 \
    --results-dir results --no-nv --use-host-data \
    SonicTheHedgehog-Genesis GreenHillZone.Act1

2) With GPU support (based on Nvidia-Docker):

retro-contest run --agent $DOCKER_REGISTRY/agent:v1 \
    --results-dir results --use-host-data \
    SonicTheHedgehog-Genesis GreenHillZone.Act1

Step 6.5: Upload your Docker:

docker push $DOCKER_REGISTRY/agent:v1

Step 6.6: Start a new job on OpenAI contest website

After we uploaded our docker, we can start a new job:

Submit! Then All is done!

4 Conclusion

Above is a summary to prepare some dirty works to do the contest. Hope it is helpful!