So, you want to become a Self-Driving Car Engineer?

14 min readJan 10, 2017

I gathered a bunch of information from several job posts for Self-Driving Car Engineers and wanted to share what I found to be the most common skills needed for the job.

I’ll be going through each one of the skills, tools, libraries, algorithms, etc that I found and I’ll describe each one of those in a little more detail.

Fundamentals

Lots of job post actually ask for fundamental skills and not just specialized skills. Surely, you have more recruiters asking for TensorFlow than Linear Algebra, but several others actually explicitly ask you be comfortable with Machine Learning fundamentals.

Here is a list of some of the fundamentals that caught my attention the most and a quick analysis as to why they are required, and how you might actually be leveraging those skills on your position as a Self-Driving Car Engineer.

Linear Algebra

This was one of the top if not the top fundamental skill they want you to know. Linear Algebra is how Machine Learning is sped up. Without vectors, matrices, and tensor, you might find yourself with very inefficient for loops and unacceptably long compute time. Sure, most likely you will be using libraries to leverage matrices instead of coding these from scratch. However, if you don’t know how to multiply matrices and vectors, you are definitely in disadvantage.

Calculus

Calculus? Well, derivatives are essential to arguably one of the most popular Machine Learning algorithms out there, Gradient Descent. Knowing calculus allows you to clearly understand optimization techniques, and knowing these allow you to better design, debug, and troubleshoot your implementations.

Mathematical Skills

I read a lot of Quora questions asking if you need to learn math in order to become a Machine Learning Engineer, and to me it seems rather a funny question. Why? You may ask. Well, I really think you don’t need to learn math if you just want to be able to build one or two predictive models, but you definitely need it to be a world-class engineer. I can think of an analogy with music, for example, you definitely do not need to learn to read musical notation if you want to play the guitar and even to be famous, however, most of the world-class musicians, and I’m talking about those who compose and play good and complex music, do know to read their score sheets. Think about it this way, Self-Driving Car problems need a symphony, to be part of a Concert Band you must be able to read notation, make sure you get your mathematical skills brushed up.

Probability Theory and Statistics

These two are often mixed up, but they are different and both important. Statistics concerned with understanding an underlying processes from observations while probability theory is concerned with the underlying process and this might create new observations. Also, statistics allow you to characterized the confidence of your predictions.

On the job posts I read, these two are needed, though probability being slightly more prominent. However, how important you think the confidence of your predictions are when dealing with transporting lives? Some posts explicitly wanted you to be able to do data analysis and visualization on off-line data, however, using statistics to make better prediction is something to consider.

Input Sources

Self-Driving Cars collect data from multiple sources. Almost all job posts want you to be familiarized with these input sources and their data. Some of these types of data are hard to collect, such as LIDAR and RADAR, some are readily available if you have a smartphone, E.g. Camera and GPS. Make sure you check data from all of these sources or at least understand what these are and what kind of information they collect.

Camera

This was virtually in every single job post I read. You must know how to handle batch of images. Self-Driving Cars use data from this source for a number of things, for example, identifying pedestrians, classifying traffic signs, localizing lane lines, etc.

LIDAR

It stands for “Light Detection and Ranging” and it is roughly a distance sensor. It works by emitting a pulse of light and measuring the time taken for its reflection. It’s common to see arrays of around 64 pulse emitting units in a single LIDAR system allowing the creation of 3D images. LIDAR in conjunction with other sensor like GPS and IMU can give us accurate picture of the object around us. LIDARs can be very expensive increasing the cost of self-driving cars considerably. This is one of the reasons why the boom of deep learning and its ability to extract more information from cameras is so important to this field. However, LIDAR is still very common in job posts out there, so make sure to get some LIDAR data an explore it in depth.

RADAR

RADAR and LIDAR are very similar, at least, in their purpose. RADAR stands for “Radio Detection and Ranging”. Unlike LIDAR, RADAR uses radio waves instead of light, this makes RADAR slightly more difficult to direct. You can think of RADAR as sending UDP packets and LIDAR as sending TCP packets. Moreover, RADARs are use also to detect the proximity of vehicles approaching the car from anywhere.

GPS

GPS stands for “Global Positioning System”, and this sensor does that really well. But wait, it can precisely tell you your positioning in the globe, within 10 meters or so. Although this is already a very hard job (position an object anywhere in Earth), this is just unacceptable for a self-driving car. 10 meters would get you off the cliff in that

sharp turn on your way back home. In self-driving car technologies, however, GPSis just one more sensor, and there are other measurements and pre-loaded data that when fused together show a clear and accurate picture of your environment.

IMU

It stands for “Inertial Measurement Unit”. Is an array of sensor like gyroscopes, accelerometer that give a picture of the forces that the object is exerting. For example, with an IMU, you can get the acceleration, inclination, angular velocity, etc, of an object. In the self-driving car world, IMU are useful to fuse this data with LIDAR, GPS, and get a more accurate positioning, velocity and so on.

Fields

There are several fields that you should familiarize yourself with before you go to a self-driving car interview, some are rather obvious, some you might never heard of before. In this list, make sure you get to at least know what each of these fields are concerned with.

Artificial Intelligence

AI could be seen as the mother of all computer intelligence. The main umbrella that covers all the newest fields like Deep Learning is Artificial Intelligence. Wikipedia describes AI as “intelligence exhibited by machines.” By this definition, non-intelligent agent that exhibit intelligence are also consider AI agents. For example, a chess playing agent does not necessarily know what to play through intelligence, but instead it uses simple data structures and search algorithms to find the best possible move given the current table state. This, to us humans seems like, intelligence, but to computer it is just raw processing power. Also, things like Knowledge-Based AI, for example, also can exhibit “intelligence”, though this intelligence is proportional to the amount of intelligence described to the system by human experts. The “cool” side of AI comes withing the Machine Learning field which is a sub-field of artificial intelligence. Regardless, make sure you get your basics down in machine learning. Things like statistics, search, Bayes rule, etc are good to be aware of.

Machine Learning

Chances are, if you are reading this far into this post, you at least have a solid understanding of what Machine Learning is. Almost every job post I read asked for knowledge in this field, lots of them directly. Make sure you know what the core of machine learning is, however. It is not enough to know a definition, but instead understand what things like over-fitting, hyper-parameters, validation sets, bias, variance, cross validation are. This come really handy in your time as a Self-Driving Car Engineer. You will need, for example, to know how good a model is, try to find better models, split and use data in the most effective manner and so on. All of these things are taught in Machine Learning, make sure not only to know but to preach these things and you will be a much stronger candidate when the time comes.

Computer Vision

Computer Vision is concern with gaining an understanding from images. This includes video images which is nothing but a sequence of images. Computer Vision is a interdisciplinary field in which scientists that study vision in humans and scientists that study vision in artificial systems contribute new discoveries. Computer Vision involves image manipulation, and advanced techniques like HOG transform, etc. However, one of the biggest contributors to the Computer Vision comes from the Deep Learning field which is the field were most of the recent top performances in the ImageNet competition come from. Still, in order to design better Deep Learning models you must gain offline understanding of the images your system is likely to encounter, and then design models robust enough to consistently solve the problem you present it.

Deep Learning

While back in the day, Computer Vision was concern with creating features that could be more easily interpreted by Machine Learning models to create accurate predictions, Deep Learning revolutionized this paradigm by compiling Computer Vision and Machine Learning into a single field. Deep Learning is able to classify images from raw pixels through the use of convolutions and deep neural networks. The birth of deep learning can be tagged many years ago, however, only with the recent explosion of data, and the improving capacity and speed of GPUs, deep learning has been able to reach above human performance in vision. Deep Learning has a bright future ahead, and one of the promising outcomes of it will probably come from the merging of Deep Learning and Control Theory. There are a few high-caliber researchers looking into combining these two to solve problems in robotics and the results are astonishing. Make sure to do your homework on these.

Control Theory

Although more of an Engineering field than Computer Science, Control is important in any close-loop system. Since at least the early 1970’s, control theory is dealing with maximizing the return from (or minimizing the cost of) the operation of physical, social, and economic processes. Control Theory involves things like trajectory optimization which is the process of finding best performance while satisfying given constraints. Trajectory optimazation has an benefit over Supervised Learning, which is that you are able to deal with complex dynamics without worrying about policies. Supervised Learning on the other hand, is great at finding complex policies but has trouble dealing with complex dynamics. These two combined promise to revolutionize control systems for the years to come.

Reinforcement Learning

Reinforcement Learning is the field in which many of the algorithms for sequential making reside. Control Theory and Reinforcement Learning are somewhat related, though one being more of an engineering field while the other a computer science field. Both are important to know since both deal with sequential decision making. From these fields is also important to know what Markov Decision Processes are, in particular, there were a few mentions to Partially Observable Markov Decision Processes (POMDPs). MDPs are a way of representing complex systems. There are many generalizations of MDPs, one of them being POMDPs is concern with environments that are not seen in totality by the agent. It is a safer bet to hold a probability distribution of the possible states of the environments, than mistakenly assume the agent is on a state. POMDPs seem to be of importance to self-driving cars. Make sure to read up on them.

Signal Processing

Signal processing is a branch of electrical engineering concerned with understanding data streams. In self-driving cars, there are multiple sensors sending measurements of the environment multiple times a second. All of the data from these sensors must be processed in other to find meaning on it. Even an image could be considered an image if we considered the change of the color channels in space instead of time. A video stream is a signal that varies in both space and time. signal processing is commonly used to reduced noise, to correct distortions and more generally to extract information from input sources.

Techniques

In this section, I grouped some of those techniques that are at the heart of fields mentioned above. By technique I mean the steps of carrying out a particular task, but not necessarily a single algorithm. It’s important to note that all of the techniques mentioned below actually show up in the job descriptions and it is not just a list created from the fields above.

Motion Planning

Motion Planning can be briefly defined as the process of breaking down a desired motion task into small actionable control commands. There are several important algortihms that cover motion planning, one of the most popular is A*. In the self-driving car world the whole point is transportation, in other words, get someone of something from point A to point B. Motion Planning tasks are at the heart of this technology.

SLAM

It stands for “Simultaneous Localization and Mapping”, and it is concerned about building a map of an unknown environment while simultaneously keeping track of the agent’s localization within that map. The beauty of SLAM is that it really doesn’t depend in any given configuration, but instead it uses any series of sensor observations to improve the accuracy of its prediction. SLAM is based on Bayes rule since it updates the location posterior given the observations and a map, and simultaneously updates the map posterior given location and observations. SLAM can be thought of an Expectation-Maximization like algorithm in which the algorithm iterates between prediction and sensing improving its accuracy with each iteration. The most common algorithms under SLAM are Kalman Filters and Particle Filters. Lots of the job posts I came across asked for either mapping or localization, and many of those ask for both and more specifically for SLAM. Make sure to brush up in other mapping and localization algorithms. I go over a few of those on the following section.

Image Processing

Image processing is the modification of images using mathematical operations. These operations usually come in the form of matrix operations, that being the reason why Linear Algebra is an important foundation to us. There are many ways to process an image, some of them include normalization, sharpening, blurring, crop, shift, rotate, flip, and so on.

Sensor Calibration

Sure, sensors usually come with an accuracy of 0.000000001 out of the box. But that is not always as good as it looks. In reality, sensor often need to be calibrated to ensure maximum accuracy. For example, cameras add distortion to images in which, for example, “objects may appear closer than they are”. Also, other sensor may be prone to ambient light, etc. Inertia sensor are sensitive to alignment, so poor installation might give inaccurate readings. Overall, since as a Self-Driving Car Engineer you will be dealing with multiple sensors, you will want to familiarize yourself with calibrating each of them.

Sensor Fusion

Sensor Fusion is a technique for merging data from different sensor sources in an attempt to reduce uncertainty and improve measurement accuracy. These sensors don’t have to measure the same variables, sensor fusion is concerned with using all sensor data regardless of how homogeneous the set of sensors are. You can think of some ways of combining LIDAR, RADAR and Camera data to accurately determine the distance between a car and the autonomous car. One interesting application of sensor fusion however, can be to used to determine crowded the road is by using acoustic data and images.

Feature Extraction

Feature Extraction is the process of deriving values from measured data to be used in Machine Learning algorithms. We can imagine feature extraction in the context of images as obtaining basic shapes from the images, like circles for the eyes, nose, etc.

Object Detection, Classification & Tracking

These are probably complex techniques deserving a section of their own. There are 3 common problem in Computer Vision, (1) to detect objects in an image, this can also be thought of image segmentation or partitioning, (2) to classify the objects in an image, one thing is to say there is this yellowish blob in the image an another to identify the lion in front of you, classification is an important aspect of Computer Vision, (3) to track objects in a video stream, also known as Video Tracking. As we can imagine as a Self-Driving Car Engineer you should be able to deal with all three; detecting, classifying, and tracking objects in images and videos, pedestrians, trucks, other cars, etc.

Algorithms

Lots of algorithms were mentioned, some job postings only show things like advanced algorithms, search algorithms, graph algorithms or even complex data structures, however, I compiled a list of the top algorithms that I found related to the fields and techniques mentioned above.

Kalman Filters
EKF
Bayesian Networks
SLAM
Yamauchi
STC, CCD, A*, D*
Particle Filters
Other Bayesian Filters

Languages

This one was a little easier than I thought it would be, the top programming languages hand-down were C++ and Python so make sure you brush up your memory handling skills. Additionally, there a couple of job postings that asked for a couple other languages, Java and Matlab were the top 3rd language.

C/C++
Python
Java
Matlab/Simulink

Libraries

Out of the big problems that you have solved in your life, have you solved a single one all on your own? Please, say no. The truth is, leveraging libraries is very important when working as a Self-Driving Car Enginner. I compiled a list of the libraries that are mentioned the most:

OpenCV
TensorFlow
Torch
Theano
Scikit-Learn
CUDA

Tools

“Give us the tools, and we will finish the job” said Winston Churchill in 1941 in the middle of World War II. What is a man without them? In this section I share some of the common tools that I read on the postings. Note that Hadoop is not necessarilly a tool, but a collection of them. Still, I added the set of ‘distributed tools’ and perhaps libraries that were mentioned.

Linux Environment
ROS [ROS2] Middleware and Tools
gbd/ddd (debuggers)
git
Jenkins, Travis
Docker (Ansible)
Distributed tools (Hadoop, Spark, Storm, Kafka, Akka, Zookeeper)
GIS Software (Point Cloud)

Software Engineering Methodologies

It is common in job postings from large companies to have some of the common software development and engineering buzzwords in their listings. Interestingly, the more true tech giant the company, the less they ask for these things. Still, I add these here so you get at least a quick reading to some of these posts.

Unit Testing, validation & release (development and product lifecycle)
System & Customer acceptance testing
Object-Oriented Analysis
Object-Oriented Design
System Integration
Software Architectures
Monitoring and debugging

Software Development Skills

These skills are somewhat in the “Software Engineering”, but are more development-related.

Code Optimization
Modular Code
Implement & Adapt Complex Algorithms (E.g. from literature, from other teams, etc)
Computer Networking
Data Collection
Offline data processing and analytics
Large Code base manipulation & management
Robust, Safety-critical, efficient code

Deployment Environments

A lot of mentions to being familiar with distributed and parallel architecture. For example, make sure you train and deploy a Neural Network across different GPUs, or even servers with multiple GPUs.

Distributed Architectures
Parallel Architectures (GPU)
Scalable Architectures
Real-Time Operating Systems (low-latency)
Embedded Systems
[Embedded] Firmware
High Availability
Fault-tolerance

Personal Skills

Next, I collected a set of “personal” buzzwords some of the employers are looking for:

Inquisitive
Proactive
Interested in learning
Work Independently without supervision
Take Initiative to suggest and implement improvements
Escalate risk
Work under aggressive schedules towards world-class solutions
Talented, energetic & committed
Ability to deliver exceptional results in challenging tasks
Fearless
Thrives under pressure
Hyper focused
Yearn for cutting edge and high profile projects
Passionate about their craft

Interpersonal Skills

Finally, great that you are ambitious, but can you communicate with others? Here is a list of interpersonal skills mentioned throughout.

String verbal and written communication skills
Collaborate in multi-disciplinary team
Fast-paced & dynamic environments
Escalate risks
Mentor-ship
Working remote (time-zones) & distributed (location) teams
Open-Source contributor
Passionate about working on startups
Leadership, entrepreneurship & persuasion

Final Thoughts

As you navigate through this list, have in mind that this list was meant to be more like a general, wide view into the skills needed to become a Self-Driving Car Engineer. That being said, as important as knowing a little bit of everything, is knowing a lot about one thing. You must be able to pick a single thing in which to become an expert. Look into yourself, connect with your passion, and then prioritize the skills that you will study in your journey towards your dream job.

Last but not least, if you are truly interested in becoming a Self-Driving Car Engineer, make sure to checkout Udacity’s Self-Driving Car Nanodegree program. I’m about to complete the first term, and I would give very high ratings to the program. I’ll make sure to review the first term once it ends.

If you like to gather more information regarding becoming a Self-Driving Car Engineer, I recommend you talk a look at David Silver’s Medium posts. In particular the following were very interesting:

References

Paysa.com: Programming Languages, Libraries, Algorithms, etc. Take a look for yourself.
Indeed.com: Job postings from companies like Bosch, Waymo, Lookheed Martin, Drive.ai, HERE, NVIDIA, DCS Corp.
Wikipedia.com: Getting definitions of some of the fields, etc.