Google football environment — installation and Training RL agent on Intel DevCloud
Co-Author: Vishal Bidawatka
Google brain team released an open source football environment for the purpose of reinforcement learning research some days back. They have provided the base code (Github) and also their research paper on it.
Understanding the environment
Let us break down this complex environment and understand its state space representation, action space and other configurations required for successful RL model training-
State space
This environment supports three types of state representation.
- Pixels: The representation consists of a 1280×720 RGB image corresponding to the rendered screen. Note this includes both the scoreboard and a small map(minimap) in the bottom middle part of the frame from which the position of all players can, in principle, be inferred.
2. Super Mini Map ( SMM): SMM is basically a stack of binary matrices defining the mini-map rendered at centre-bottom of the screen. The SMM representation consists of four 96 × 72 matrices encoding information about the home team, the away team, the ball, and the active player respectively. The encoding is binary, representing whether there is a player, ball, or active player in the corresponding coordinate, or not.
3. Floats: The floats representation provides a more compact encoding. It consists of a 115-dimensional vector summarizing many aspects of the game, such as players coordinates, ball possession and direction, active player, or game mode.
Note — Only pixels and SMM representation can be stacked across multiple consecutive time-steps (for instance, to determine the ball direction).
Action space
The action space of gfootball is quite interesting and quite co-related with the actual football game. The action is given to the currently active player i.e the player with a blue bar on his head. The environment supports 20 actions.
They have already implemented built-in rule-based bot for inactive players that follows actions corresponding to reasonable football actions and strategies, such as running towards the ball when we are not in possession or move forward together with our active player.
Scenarios
Google Brain team has provided different scenarios of varying difficulty. Its main goal is to allow researchers to get started on new research ideas quickly, and iterate on them. The easiest scenario for the agent to learn is “empty goal close” in which only one player is standing in front of goal without a goalkeeper.
Installation
Let’s start setting this cool environment into our system. We are using system having ubuntu 18.04 operating system and python 3.6. The best and easiest way to install the game engine and the library is by cloning the git repository.
Note — It is preferable to create a anaconda environment or python virtual environment first and then install dependencies in that environment.
- git clone https://github.com/google-research/football.git
- cd football
- pip3 install .[tf_cpu] --process-dependency-links
Or for gpu version
- pip3 install .[tf_gpu] --process-dependency-links
- This command can run for a couple of minutes, as it compiles the C++ environment in the background.After installation, you can test your installation by running this file environment_test.py from our GitHub repository.
- git clone https://github.com/Ujwal2910/Deep-RL-on-Gfootabll-Google-football-OpenAI-style-environment.git
- cd Deep-RL-on-Gfootabll-Google-football-OpenAI-style-environment/
- cd enviornment_render/
- python3 environment_test.pyIt will continuously render the frames:
If you encounter numpy version error the root cause may be the overlapping numpy version. Just try uninstalling numpy and try running the script again.
Intel Devcloud
We have used Intel Devcloud for training our deep learning script and later on for processing simulator frames. Not only that using Intel Distribution for Python 3.6 we were able to achieve faster application performance in comparison to normally running the code on our laptops.
This software accelerates Python performance even on anaconda by supercharging python applications and speeds up core computational packages.
We learned how this distribution could actually help us in utilising the intel architectures for running tensorflow for CPU even faster. Since we were using NumPy and SciPy, the intel data analytics acceleration library did give us a boost in processing frames of the simulator.
You can download the Intel accelerator software for boosting these libraries from here. To read more about Intel’s Devcloud and architectures refer here.
Things to remember
We have currently reported two issues to google football Github repository and they have acknowledged it and gave a temporary solution until they fix it.
- In float representation that is 115–dimension vector, the 115 dimensions is only specified for the 11v11 scenario, for all other, this number varies. So you can first check with state.shape[0] and then build your model accordingly. ( link to issue )
- Currently, their program changes the present working directory of the running code so while saving the models use absolute paths. ( link to issue )
The repository will include implementation of different RL algorithms on different scenarios and you can follow by referring here-
https://github.com/Ujwal2910/Deep-RL-on-Gfootabll-Google-football-OpenAI-style-environment
Do star and fork the repository and wait for upcoming training results on different scenarios of this environment with state of the art RL algorithms.
