Introduction to the CARLA simulator: training a neural network to control a car (Part 1)

Maciek Dziubiński
Nov 27, 2018 · 9 min read

Intro to CARLA + simple controller for collecting data + training a neural network for steer control.

Introduction: the CARLA simulator is a platform for testing out algorithms for autonomous vehicles.
I wanted to check out CARLA, build a simple controller for following a predefined path around the track to gather data, and then train a neural network model using that data.
Approach: I used a simple CNN architecture and modeled the steering angle.
the model is able to drive the car around the track by controlling steering.


CARLA is an open-source simulator built on top of the Unreal Engine 4 (UE4) gaming engine, with additional materials and features providing:

But more importantly, CARLA is thriving; it has a great community, it’s documented well, and it’s (subjectively) easy to use.

Unreal Engine 4

But something I didn’t anticipate is that UE4 allows for developing quite advanced features without the necessity of implementing them in C++. For that, UE4 has a dedicated visual scripting language called Blueprints. I’ve used Blueprints for the creation of the race track (more details further).

Alternatives to CARLA

And two simpler ones:

In the end, I only considered CARLA and AirSim since I wanted to work with UE4, and chose CARLA because it seemed easier to contact with the community via their discord chat.

The race track

The track is composed of RoadMesh elements that look like this:

But to be able to create a smooth track I had to incorporate this element into a blueprint script called Track Generator. The race track is then created like this:

The race track on which I’ve actually rode looks like this:

I wanted to have sections of: tight corners, standard corners, and a couple of long straights. But, full disclosure, I have no idea what makes a track better or worse. For my purposes, this race track was perfect.

An interlude: a tour of the project

The two most important scripts are: and, where the first one is used for interfacing with CARLA, for controlling the car and for gathering data, whereas the second is used for training neural networks on the collected data.

Before running the script, remember to first initialize CARLA. On my system it requires the following:

cd ~/CARLA
./ /Game/Maps/Racetrack01 -windowed -carla-server

After which, to control the car using a proportion-derivative (PD) controller (more details below), I call:

python3 \
--speed 45 \
--controlled_name pd

And the result should look more or less like this:

We’ll get to the script in a minute.

The simplest possible controller

To reach my goal (training a neural network capable of going around the track), I started with the simplest controller that would collect the data for me. And it doesn’t get much simpler than the PD controller:

in which the steering angle at a given time t depends on the Cross-Track Error (CTE), and the time derivative of the CTE. The CTE is the distance between the center of mass of the car, and the desired trajectory. Also, the controller needs to be calibrated, i.e. it’s up to us to choose the coefficients kp and kd such that the car drives (relatively) smoothly. (The fact that Medium doesn’t have support for LaTeX is a pain to say the least.)

Admittedly, this is not the simplest possible controller because, as you might have thought, one might try a “P only” controller that ignores the second component. And yes, this is an even simpler controller but the dCTE/dt component plays an important role of stabilizing the output. Without it, the controller quickly becomes erratic:

so it wouldn’t get the car around the track.


In the first iteration of this project I didn’t really care for velocity, I just wanted the car to go forward; not too fast, not too slow. To that end, I used a simple clip_throttle function that keeps the throttle within reasonable bounds, and tries to keep the speed close to the target_speed:

Notice that this results in a wide range of throttle values, which in turn results in high variance of the velocity. But that’s OK; my reasoning was that this “augments” the data set and yields a more generalized model.

Technically, you could take the time and build a separate PD controller for the throttle so that a target speed is kept fairly constant. But this would require its own (kp, kd) parameters, making the model slightly more complicated.

But more importantly, in the second blog post we’ll use a controller that controls both actuators, steering angle and throttle, simultaneously. This approach is called Model Predictive Control (MPC), and is admittedly more complicated, but it provides a trajectory that is much more efficient. And if this didn’t whet your appetite for the second blog post, here’s a video illustrating how an MPC-controlled car drives around the track:

Right away you can notice that the frame rate is much lower. This is because MPC requires some heavy optimization to be done on the fly, and my implementation of this controller is written entirely in Python.

Building a neural network

Admittedly, it’s pretty pixelated and a bit unclear. But note that this representation carries 3D information that is essential for driving the car along the track.

I used keras for training the neural network, and used a fixed, singular architecture. I prefer raw code instead of fancy visualization tools, so here’s the definition of the model:

I didn’t spend much time tweaking the architecture or the optimization parameters. I chose “reasonable” hyperparameters, and instead focused on modifying the input and adding auxiliary losses. But more on that in the second blog post.

The model had one job: predict the steering angle for a given depth map frame. It didn’t make much sense to predict throttle because it was independent of what was in front of the car (at least for our PD controller; this will change in the second blog post). So the throttle was also modulated using the clip_throttle function.

To train the model, I ran:

python3 -c pd

I had about 90k frames and the net took about 2h to train, but that’s because: 1) the model was over-the-top-complex, 2) I didn’t shrink the input image, and 3) the default number of epochs was 50. However, the MSE on the test set quickly became flat:

And in reality, starting with “Epoch 2” the scatter plot of predicted vs actual steering angles looks pretty much constant:

This is how the model with the lowest MSE on test set managed to drive around the track:

I think the coolest thing is that the neural network model seems to stabilize the oscillations better than the PD controller.


Also, I’ve noticed that models with the lowest MSE failed to steer the car out of tight turns. That is, high (but rare) values of the steering angle were never produced by such models in the simulator (even if on the test set the model seemed capable of yielding those steering angles). So I’m inclined to evaluate the performance of the model in two types of scenarios: normal, and extreme situations. But that’s for the future.

Introduction to the CARLA simulator: training a neural network to control a car (Part 2)

Training a neural network for driving an autonomous RC car


Race track: For building the race track I followed this tutorial: youtube link. Also, I found this part from the UE4 documentation valuable. Actually, the whole UE4 is documented very well, here’s one of the gems I intend to explore.

If you enjoyed this post, please hit the clap button below and follow our publication for more interesting articles about ML & AI.

Acta Schola Automata Polonica

Scientific blog about artificial intelligence and machine…

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store