How to record data for reinforcement learning agent from any game on Linux
Written by Filip Nabrdalik, Piotr Smuda and Piotr Tempczyk
Number of games supporting OpenAI Gym API is already big and is still growing. These games enable you to train reinforcement learning agents with ease.
But what, if you want to play your favorite game and it hasn’t got an API suitable for RL? If the game has any API, your problem is partially solved but when the game of your choice is meant to be played only using keyboard, mouse, pad or racing wheel your problem is much worse. From this blog post, you are going to find out what you can do in this situation.
As an example game, we are going to use F1 2017, which you can get for free from here.
Solution
Recording frames
First of all, we need visual input for our reinforcement learning algorithm. We can do this using some Linux tools.
We can achieve this using only the command line. To record a window on the screen we are going to use ffmpeg and built-in Xorg utilities. To install ffmpeg in Ubuntu, all you need is type in the command line the following command:
sudo apt install ffpmeg
At first, let’s have a look at the example ffmpeg command which allows us to capture 800x600 video with upper left corner located at position (50,100) on the screen.
ffmpeg -y -video_size 800x600 -framerate 30 -f x11grab -i :0.0+50,100 -c:v libx264 -crf 0 -preset ultrafast f1_recording.mkvWe use the x11grab input video device to get the screen output, then we process it and save it as MKV file using the x264 codec. We need to give a proper input to the ffmpeg command, which has to have the following format
-i :0.0+50,100which can be translated as:
-i :DISPLAYID.SCREENID+XPOSITION,YPOSITIONLet’s see how to get the information about a window. First, let’s input w -hs command and find out the display identifier (FROM column). You can also find it in the DISPLAY env variable in your shell.
01:07:38 up 5 days, 58 min, 4 users, load average: 0.40, 0.37, 0.41
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
oli tty7 :0 Sat00 5days 4:22m 0.94s gnome-session
oli pts/4 :0 Sat00 47:09m 0.77s 0.77s /bin/bash
oli pts/6 :0 Wed02 0.00s 0.12s 0.00s wThe second step is to execute xmwininfo -tree -root and find out the window id. Example output:
Root window id: 0x29a (the root window) (has no name)
Parent window id: 0x0 (none)
69 children:
0x2800001 "/usr/lib/firefox/firefox": 10x10+10+10 +10+10
0x220003c "F1 ": 190x29+487+105 +487+105
...Now we can get detailed info about the window which we are interested in by executing xwininfo -id 0x220005c
Absolute upper-left X: 0
Absolute upper-left Y: 0
Relative upper-left X: 10
Relative upper-left Y: 38
Width: 800
Height: 600
Depth: 24
Visual: 0x21
...We have got all the information we need to record. Let’s start ffmpeg in a separate terminal.
ffmpeg -y -video_size 800x600 -framerate 30 -f x11grab -i :0.0+50,100 -c:v libx264 -crf 0 -preset ultrafast f1_recording.mkvAfter pressing CTRL+c the recording is stopped and we can play the file using any video player. We can also use pipe to play the video in VLC Player or process it with another tool.
ffmpeg -y -video_size 800x600 -framerate 30 -f x11grab -i 0.0+50,100 -c:v libx264 -vf -crf 0 -preset ultrafast -f mpegts pipe:1 | vlc -Notice that the file format is different when using a pipe. You can play with different options available in ffmpeg to achieve different results. Have a look at ffmpeg documentation to find out many options which this tool provides. You can also check out our repository to see bash scripts that we used during the early stages of our project.
Recording with Python script
We have created a simple python script which automates the whole process. You can find it here. Just change the APP_NAME variable to record the app you want. It was written using ffmpeg-python library which nicely wraps around ffmpeg. The library is quite powerful and you can use it to process your video with numpy or tensorflow.
In Python code, we follow exactly the same steps which were described in the previous chapter. First, we get the display id with get_current_display_id(), then we search for app window id that we need using a part of its name get_window_id(name), the third step is to get window position and size with get_window_attrs(window_id). Finally, we can pass this information to the ffmpeg-python library and record what we need. In the example below, we also do a little bit of processing before saving it to a file. We crop and re-scale the input from x11grab, so just remove corresponding lines from ffmpeg command builder if you don’t want to do so. Also, remember to check out our repository.
Getting in-game data and controller’s state
F1 2017 game allows to extract a lot of game data, where the most interesting is car’s telemetry, which can be obtained through User Datagram Protocol (UDP). Moreover, using external software it is possible to get game controller’s state (in our case it was Logitech G27 wheel), which in combination with telemetry gives lots of useful information.
To enable telemetry output you need to go to the game options and navigate to Game Options -> Settings -> UDP Telemetry Settings and turn on UDP and broadcast options. You also should configure the IP address, port for the receiving application and send rate per second, which we set respectively as 127.0.0.1, 20778 and 30. So, from now the game will be sending in-game data during racing such as throttle’s/steer’s state or race position! More about available data you can find here.
The only thing left is to receive it using Python. First of all, it is worth pointing out that we use pre-installed socket module for telemetry data and external pygame library for wheel’s data. Therefore, it is possible that you need to install pygame — just type in the command line following command:
pip install pygame
Now, let’s look at how all works. Firstly, we need to initialize socket and getter, which enable us to receive respectively telemetry and wheel’s data (for both, list of data fields’ names should be provided in the correct order!). For setting socket we use the same IP address and port which are set in game:
SOCKET = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
SOCKET.bind((UDP_IP, UDP_PORT))and for the getter it looks like this:
pygame.init()
wheel = pygame.joystick.Joystick(wheel_index)
wheel.init()where wheel_index is an index of the controller in system (usually it is 0 if there is only a wheel connected to PC as controller).
In the F1 2017 game, data is broadcasted only when you are in control of the car (if game is paused or in the menu, there is no available data). So, we can try to capture data in loop using our socket and getter. If there is any telemetry data, we intercept it from the buffer and decode it from bytes to numerical values using the struct module:
telemetry_data, addr = SOCKET.recvfrom(1289)
value = struct.unpack(value_type, telemetry_data[x : x + nbytes])[0]where 1289 is number of bytes in packet size, value_type stands for “f “ or “b” respectively for float or byte values, x is position in the packet and nbytes is number of values bytes. At the same time, we take wheel state with get() method of WheelGetter class and combining it with telemetry data we write all data to one row of pandas.DataFrame. Finally, we can save received data to the new line in CSV file on disk. To find out more about implementation please look at our repository.
Final remarks
This blogpost is one of the results of a bigger project, which goal was to train an RL agent to play F1 2017. There were many other people involved in this project: Maciej Śliwowski, Bartosz Topolski, Tomasz Malisz, Jakub Grudzień, Tomasz Arczewski. During this project, we also developed a framework for reinforcement learning named People’s Reinforcement Learning (PRL).
If you enjoyed this post, please hit the clap button below and follow our publication for more interesting articles about ML & AI.
