Experience Matters: Our Time Developing Autonomous Vehicles
Raj Nair, executive vice president, Product Development, and chief technical officer, Ford Motor Company
At Ford, we take our commitment to autonomous vehicles very seriously.
We started more than a decade ago, at the beginning of the modern-day race to develop autonomous vehicles, and we are at the forefront today. But Rome wasn’t built in a day, and neither will be autonomous vehicles. It takes hard work and hard-earned experience — experience that goes back more than a century.
It also takes unrelenting determination, as demonstrated by several members of our team who helped form our first autonomous vehicle research effort back in 2004 — and are still part of the team today. For instance, Jim McBride, who would not give up on the dream of delivering a fully autonomous vehicle, even when it seemed more like science fiction than actual science to some.
It takes support from those like Jeff Rupp, who, as former head of Ford advanced safety technologies, foresaw a world where accidents and fatalities due to human driving errors are greatly reduced. According to the National Highway Traffic Safety Administration, 94 percent of traffic crashes involve a mistake by the driver. So, if humans are the weak link, why not try to build a system to support or replace them?
And it takes knowledge and outside-the-box thinking from those like Jerome Ivan and Doug Rhode, two engineers reimagining what it will take for a vehicle to be capable of fully autonomous driving.
At Ford, the story of bringing autonomous vehicles to the world is still being written. Yet the journey has already been a rich one, and one best told by the team who has lived it from the beginning.
The beginning
Jim McBride, senior technical leader, Autonomous Vehicle Development, Research and Advanced Engineering, Ford Motor Company
When we heard about the DARPA Grand Challenge, Jeff Rupp and I decided to serve as industry volunteers for the event because we thought it might be relevant to automotive safety and we wanted to check out the latest in robotics technology and capability. When we got to the competition in 2004, we knew we were part of something big. We met so many leaders in this emerging field: government officials, robotics experts from various universities, and engineers from technology companies, the defense industry and national labs.
The influence this event had on the automotive industry can be summed up in the story of how I met Dave Hall, who was there competing. We shared ideas about how to approach autonomous driving. Dave was using a home-built, stereo-vision sensing set, which I told him I felt was insufficient alone to drive fully autonomously. I told him that LiDAR, in addition to machine vision and radar, was essential, as we needed to have a live 3D representation of the world surrounding the vehicle. My suggestion was to glue a bunch of lasers together and gather a larger field of view to form a more comprehensive image.
Dave returned home from the DARPA challenge and began building a prototype based on our discussions. One of the first 360-degree LiDAR sensors he developed was provided to Ford in 2006, and today it still sits atop our first-generation autonomous vehicle — a tough, self-driving Ford F-250 Super Duty pickup. I’m proud to say Dave’s company, Velodyne, has been Ford’s supplier of LiDAR sensors for all three generations of our autonomous research vehicles.
Ultimately, the first DARPA Grand Challenge became the launching pad for autonomous vehicle research at Ford. We got a firsthand look at technology that would soon shape the autonomy landscape: mapping, sensing, algorithm development and more. Inspired, we left with the sense this would be part of Ford’s future.
We decided to assemble a Ford team to compete the next year. To do so, we needed to apply and qualify, just like everyone else. So we pulled two others, Doug Rhode and Jerome Ivan, from the Ford advanced safety team, which Jeff headed. Doug and Jerome became integral to creating the electrical systems and hardware that were needed to make the car move.
Our small team worked tirelessly over the next several months, developing sensors and virtual-driver algorithms that perform like a human driver. At the time, the government was trying to inspire innovations in autonomous vehicles in an effort to remove military troops from wars in the Middle East.
We chose the F-250 Super Duty because, based on the objectives of the challenge, the vehicle had to survive the punishing desert environment. From our perspective, if our entry was going to fail, it wasn’t going to be because it couldn’t handle the terrain. As it turned out, while the challenge took place in the desert, the groomed gravel routes were not demanding enough for the off-road durability and capabilities of a Super Duty.
We entered our first challenge in 2005 with the team name Intelligent Vehicle Safety Technologies, or IVST for short. We didn’t advertise that we were Ford Motor Company since our objective was to conduct research and development, not get media coverage. But we came to find out that we were the only automaker-led entry with a vehicle that had custom-designed sensing equipment and software algorithms written by its employees.
We had limited time, so we had to get creative about problem-solving. Our custom forged-aluminum wheels arrived the same day we began field testing in Nevada, and with the oversized racing tires, they turned out to be too big to clear the wheel wells. So we removed the fenders and used plastic trash cans to salvage the field test. Custom fender flares were made as soon as we got home, and were used in all the subsequent races.
When the day of the challenge finally came, all we could do was turn our F-250 loose.
I crossed my fingers and hoped the truck, which we had spent a year putting together, would come back to us. We were extremely safe and conservative, but this was the first time we let our F-250 run with no one in it, and it was like watching a novice pilot fly solo for the first time. The truck actually got up to 50 mph, which was fast for the time.
In the end, our autonomous truck made it to the finals. We had a chance to win the race, but as our truck was capable of detecting pedestrians in its path — a feature many of the competitors didn’t have — it intentionally avoided a photographer who had stepped onto the course. The large truck had such a wide turning radius that it drove outside the roadway boundaries to avoid the photographer and was disqualified.
We were happy the truck actually responded as it was programmed to do. And even though it didn’t finish the course, which it typically did during testing, it never was damaged, nor did it damage anything or hurt anyone on the race course.
We did so well that we earned a $1 million grant from DARPA to return for the 2007 Urban Challenge. In the end, we were one of only six teams to qualify as a finalist in both the 2005 DARPA Desert Classic and 2007 Urban Finals.
Still, our work here was just beginning. We never lost sight of the end game. And now, as I think about the future, it’s our experience that’s going to make the difference in achieving our goal — putting autonomous vehicles on the road serving millions of customers.
The quest
Jeff Rupp, former manager, Active Safety, Research and Advanced Engineering, Ford Motor Company
Standing there in the desert, collaborating with and competing against the best robotics teams from companies and universities around the country, I realized we were on the cutting edge. It was truly a goosebumps moment. Everybody was trying to push the envelope — learning fast by trial and error. The whole team was energized and excited.
The day of the challenge, there was full military fanfare. Helicopters were flying overhead. It was almost surreal. We wanted to win, of course, but ultimately the goal was to learn.
After the DARPA challenges, I knew there was still a lot we didn’t know. But if we were going to stay ahead of what was sure to be the biggest thing to change the car industry since the assembly line, we needed to jump in. Over the next few years, we worked on building our advanced safety team, with Jim working on fully autonomous vehicles and a growing number of people working on advanced safety and driver-assist features.
Our autonomous vehicle research and development efforts have inspired many of today’s driver-assist features, like Blind Spot Information System, lane-keeping system, forward collision warning, and collision mitigation by braking.
This journey came with hard work and long hours. In 2007, Ford sent a car on what we called the Mother of All Road Trips across 25 states to not only test a host of new sensing technologies — radar, camera and image processing — but to log a wide range of data to be used later in simulation. This process of expanding development from driving test events to data mining and re-simulation helped deliver a new level of robustness in performance measurement and formed the basis for how we validate autonomous vehicle capability today.
It was about staying at the forefront of this self-driving revolution. For me, though, there was something deeper at stake.
I’ll never forget a newspaper photo of a car accident. The car was split into three pieces. That’s when it dawned on me, there was no crash safety system in the world that would have saved the driver.
Although catastrophic accidents like the one from that photo are extremely rare, there were 35,092 traffic fatalities in 2015. The vast majority of these were due to human error, which is why Ford believes a fully automated vehicle has the potential to enhance motor vehicle safety.
Back then, it was a small group of people working on self-driving cars. There seemed to be a mountain of insurmountable challenges. Little by little, breakthroughs in technology solved those challenges. Today, we are on the brink of having fully autonomous cars on the road. There was never a doubt in my mind that they were coming. It was just a matter of putting in the hard work to get it done.
What it takes
Jerome Ivan, design engineer, Autonomous Vehicle Development, Research and Advanced Engineering, Ford Motor Company
You could say I’ve always gravitated toward near-impossible challenges, where time was a luxury I couldn’t afford. So when Doug Rhode asked me to help on the 2005 DARPA Grand Challenge, of course I said yes.
Looking back, I was probably one of the few who had any idea how much work we were signing up for, but I was still up for it. It was up to me to design and fabricate custom mechanical hardware and electrical circuits — the parts that didn’t already exist. Basically, I had to reimagine and create by hand all the circuit boards that made the vehicle move, steer and shift — with virtually no time.
Thankfully, my training kicked in. In the early ’90s, I was responsible for chassis performance data for Ford’s racing program. Working with the race teams and experiencing action in the pits, where every second counts, you learn to work efficiently. Things move so fast, you have to react to the challenges immediately in front of you while looking farther ahead and still asking what you would do differently next time.
Back in 2007, with only a few weeks before the DARPA Urban Challenge, the team felt the software that drives the vehicle was not performing well enough to meet the expectations for the competition, and decided we needed to make a switch. When the idea was proposed to merge code from our university partners, I had little faith the task could be pulled off in such a short amount of time. But as part of the team, I lent my complete support for that decision and set to work.
Only a few weeks later, I found myself inside the truck, testing the new software for the first time at an abandoned military base in the dark of night.
My job was to take over control if the vehicle didn’t behave as predicted, but that never happened. The F-250 worked immediately, navigating around the military barracks in the dark, stopping at four-way intersections and making tight turns. I quickly glanced down at the glowing computer monitor, with one display window showing what the LiDAR was seeing — an accurate, 3D representation of the environment — and another showing the software code as it scrolled down the screen. That F-250 could see more than I could in the dark! The complexity of that vehicle, and the fact everything was working, was almost a miracle.
What made our F-250 so special in comparison to many of our competitors was that we could manually drive it anywhere, if we needed to. By simply throwing a few switches, we could also have the computers drive the truck. Most of the other competitors had to trailer their vehicles to the competition. Not us.
The Ford team was a finalist in the 2007 challenge, too, but the great benefit was the many lessons I learned, which are being integrated into the autonomous vehicles we’re building now. For instance, we used to manually start the robotic features. Each sensor and computer had an individual switch.
The written procedures to power up the F-250 were three pages long. Now, our autonomous research vehicles are turnkey.
The DARPA competition was tough, both physically and emotionally. We worked around the clock, building equipment in hotel rooms and hitting the local hardware stores for materials. But what a rush to see the vehicle drive on its own.
In it for the long run
Doug Rhode, senior technical leader, Autonomous Vehicle Development, Research and Advanced Engineering, Ford Motor Company
The first time a Ford autonomous vehicle drove itself, we were shooting a demo video to submit for entry into the DARPA competition. This was still early in the development of the vehicle, so Jerome and I were amazed when the vehicle successfully navigated a slalom — not once, but a dozen times. This was pretty remarkable for 2005.
Leading up to DARPA, I was in charge of developing all the custom software that controlled the steering, acceleration, shifting and braking functions. I worked closely with Jerome so that the vehicle was able to do any maneuver a human driver could. When we reached the 2005 DARPA Grand Challenge final, I had spent a lot of time — probably a couple thousand hours — building and testing the F-250.
When I sent it out into the desert to compete, I had the feeling you get the first time you let a son or daughter take the car out alone.
Time and time again, that F-250 showed its grit, and DARPA proved to be an interesting experience in a million ways. It taught me that putting together an autonomous vehicle requires many different skills and a balanced approach. You can have the best sensing and perception, but if you cannot control the vehicle, it’s a waste. You can have the best vehicle controls, but without the sensing and perception, it isn’t going to be successful. This translates directly to my current job, where I work with many people to build a solid vehicle platform — the parts, hardware and software that make the car go.
You have to have a tremendous number of algorithms and software to replace the driver. But at the end of the day, the technology needs a capable, dependable vehicle platform — or the car doesn’t move.
By 2013, we were ready to launch our second-generation autonomous research vehicle. We chose a Fusion Hybrid, which has an advanced electrical architecture able to generate enough power to run all the computers we put in the trunk. Of course, the goal is to remove the driver, so we needed to upgrade the powertrain, steering, throttle and brake systems to be computer-controlled. The result is an autonomous vehicle research platform that has been a very solid platform on which to develop self-driving capability.
Over the years, as the autonomous development team has grown, I’ve become much more of an advisor to our team members — all of who bring a diverse set of skills and expertise. This diversity is essential.
There are days when you achieve a milestone, and other days when things just are not working. I think of it as a roller coaster; when you are down at a low point, you just have to keep working and eventually you get to the next peak.
Autonomy will change the way people travel, and we are in it for the long run. This is serious business.
