Solving OpenAI CarRacing-v0 using Image Processing
Introduction
The OpenAI Gym is an open-source interface for developing and comparing reinforcement learning algorithms. It supports training agents to do everything from walking to playing games like Pong or Pinball. But then they only provide the environments it’s up to us, passionate learners to solve them.
I have been studying reinforcement on my own and have been playing with these environments for a while now. Here I am not going to do reinforcement learning but instead, use a classic control technique — PID controllers- to solve the car racing environment. This is just a fun and simple project I tried, out of curiosity. Hope you guys will love it.😊
Final Results 🏆
Car Racing environment 🚗
- The car racing environment comes under continuous control tasks. But there is an option to use discrete actions as well.
- The Observations we get for every step are top view images consisting of RGB, 96x96 pixels. (as shown in the figure)
- The action space is the set of triples (steering, accelerate, decelerate)
- A random track is generated for every episode.
- The reward is -0.1 every frame and +1000/N for every track tile visited, where N is the total number of tiles in track
Solution 🧾
If you are unfamiliar with PID controllers, I strongly recommend checking this awesome video.
So this is the idea — If we are able to find the centre coordinates of the track near the car’s vicinity and the positional coordinates of the car. Then we can find the difference between the centre of the track and the centre of the car. Use this difference value as the error in the PID equation.
In each observation we get, the position of the racing car’s image is fixed. {Consider the camera is following the car throughout the track}. Thus in every observation, the car’s spatial coordinates are the same, whereas the track underneath changes. Thus the positional coordinates of the racing car are easy to find!
But then how to find the centre of the track? We can find it through some sleek image processing techniques.
Let’s see how. 🌟
Image Processing 📽
Each observation from the environment looks like this
For every observation got from the environment I did the following.
- Filtered out the green part alone. So the car and the road markings are removed. To know more about colour masks check this tutorial
https://www.pyimagesearch.com/2014/08/04/opencv-python-color-detection/
2. Took grey scale of the image to reduce to just one channel, so easy to handle
3. Applied a Gaussian blur to reduce noise.
4. Applied Canny filter to detect edges
5. Cropped the portion just above the car. (2px width). And found the road midpoint
Here I have cropped the portion just in front of the racing car. From this cropped position it’s easy to get the edges of the track by finding those pixels which are non-zero. Thus we can easily find the centre of the track by taking an average of the coordinates of each edge within that cropped strip.
PID controllers 🎮
Once we have found the track centre in the vicinity of the racing car and we have found the position of the racing car. We take the difference of those values, which gives us the error for that particular timestep.
once the error is found, use that error in the PID equation. The coefficients of the equation are found manually through trial and error.
Thus for each step, we can find the steering values required to align the vehicle to the centre of the track.
Running the CarRacing-v0 env 📺
We know that the actions at every time step are inputted as a tuple- (steering, accelerate, decelerate), Here we can input the steering values that we get from the PID equation. Input the acceleration in such a manner to get a constant velocity, I gave 1s and 0s alternatively to achieve a constant velocity, you can do that using a for-loop.
Check out the full code in GitHub, It is a .ipynb file, you can directly open it in a google collab and try running it.
Try tweaking this code and get better rewards. 🏇
