Employee Spotlight: Ashesh Jain

From Cornell PhD Student to Head of Perception

Ashesh was at the hub of robotics and autonomy while working on his PhD in Computer Science at Cornell and as a visiting research scholar at Stanford AI Lab. It was around 2012, and the engineering community was quickly becoming aware that robots could be far more than humanoids helping with household chores.

Ashesh embraced this and ran with it, starting Brain4Cars and publishing an award-winning paper on Structural-RNN before graduating in 2016. He has produced an impressive number of publications covering topics like sensor-fusion and prediction using deep learning methods. He’s authored papers for NeurIPS, CVPR, ICCV, SIGKDD, ISRR, ICRA, IJRR and spoken at CVPR, ICML, RSS and a dozen other conferences worldwide. His past work has been covered by MIT Technology Review, Wired, TechCrunch, and other media outlets.

In short, we’re lucky to have him as our Head of Perception. We sat down with Ashesh and asked him a few questions about his experience going from PhD to building and leading a perception team.

To kick things off, tell us about your role at Level 5.

I manage the Perception team. Perception allows a self-driving car to understand its surroundings while on the road. We’re teaching it to recognize everything from pedestrians, to bicyclists, to the color of traffic lights. Everything you and I can do with our eyes, we have to build for the car — and it has to be in real time, and accurate to the millisecond.

We’ve got about 20 engineers on the team right now, but when I joined at the beginning of 2018, the office was nearly empty. A big part of my role, other than the technical leadership, was building this team and the pieces needed to realize our vision.

What were you doing before Level 5?

I was at another self-driving car company, Zoox, leading their tracking and sensor-fusion team (which is part of core perception).

What inspired you to join Level 5?

Aside from the exciting challenge of building my team from scratch, the vision for Level 5 and the leadership were huge factors.

I strongly believe in Lyft’s approach to autonomous vehicles and that it will truly make an impact in people’s lives. We not only have top talent, but our existing platform gives us a clear, reliable way to deliver the end product to the customer once it’s ready.

Before joining, I thought many other self-driving car companies were building great technology, but they didn’t have such a strong strategy and vision for getting it to customers. On top of this, there was a great opportunity to learn from the leaders here, which have been some of the best I’ve worked for so far in my career.

What makes the leadership great?

It’s a collection of people who are coming from different reputed companies like Dropbox, Facebook, and Google who have actually launched products from scratch (for example, the inventor of Google Street View). When someone has had experience shipping software products, they understand the process of rapid iteration. In this sense, the self-driving car is really no different: it’s simply a software that runs on a robot.

With the leadership here, I see a very product-focused, goal-oriented mindset, which helps each team work in unison even though we operate on such a complex timeline. This could not be possible if you didn’t have people who have done similar things in the past.

On a more personal level, they’ve also given me a lot of constructive feedback and I can clearly see the next step between where I am now and the next level. Because of them, I am very self-aware of my limitations and where I need to grow and improve.

What are your goals for your career at Level 5?

For me, the goal is to build a self-driving car. Everything else will be a byproduct of that. If you do great work, the rest will come.

Over the last year, what’s been the biggest challenge for you?

Working on multiple things that are equally important. To give an example, the most important thing is growing the team. The most important thing is also building the technology and hitting the milestones. Equally important is building the people relationships with other teams, because Perception interfaces closely with Hardware, Mapping, Operations, Planning — essentially all other teams. You cannot prioritize one of these over the other because they are all equally important.

We balanced these responsibilities by enabling engineers on our team to take ownership of the final deliverable, and encouraging them to interface directly with other teams to build the relationship alongside the product. A lot of good engineering, especially on a lean team, comes down to building the right culture and team dynamics. Once you have built a good team and culture, the team figures out the rest.

What has been the most impactful thing you’ve accomplished over the last year during all of this growth?

Other than building this team, I would say launching our employee pilot, which is now running in Palo Alto. Our vehicle is tackling challenging conditions on the road and is constantly getting better. This project was a huge cross-team effort and had us excited to come to work every day to meet the end-of-year deadline (which we did!). The Level 5 team is phenomenal, and it’s a privilege to work with these talented people every day.

Now that the employee pilot is launched, what’s the next big thing on your horizon?

If you think about building a self-driving car as climbing Everest, the pilot was like almost reaching Base Camp — we just got off the bus and are heading to Base Camp.

What this means technically is that we now need to build a system that is a lot more robust, and build the tools and infrastructure needed to scale the project. Right now we are “building the machines to build the machine” (i.e. all of the automation and machines needed to go into our ultimate machine: a self-driving car). Once we can scale that process, we will be ready to scale Everest.

Advice for Breaking into AV

Any advice on the technical skills needed to do well in this field?

Definitely master the fundamentals in computer science, probability, and statistics. From there, you can build on your machine learning and computer vision skills to keep on top of the trends of like deep learning and frameworks. The foundations are the most important thing.

What advice would you give to an engineer who is deciding where to have an impact in the AV field?

Try to think of not only the technology, but also the product. Technology can get you excited day-to-day, but in the long run you’ll only have an impact when you build a successful product. Especially for the self-driving car space, look at the bigger picture. It’s not just about the technology, it’s about delivering a successful product to your customer.

There is no shortage of self-driving car companies right now, but not all have a good product-market fit, so think about the technology, the people, and the product. These are the most important pillars.

How do you keep up with such a rapidly changing industry?

In some sense it’s rapidly changing, in some sense it’s not. If you actually look at what’s happening, new companies are popping up and new partnerships are forming regularly, but in a lot of cases, we won’t feel the impact of those for another three or four years.

What is rapidly changing now is the technology. Every year there are new technological developments in the space that come up that the Perception team needs to be on top of. We need to constantly be asking: “What is the state of the art?”, “How can we further improve our system?”, and “How can we take some of the things done manually in the past and replace them with machine learning?”

Lyft is great about keeping us up-to-date with the latest technology out there. We go to conferences, have reading groups, and host regular tech talks. The rest you can get from news, but I’d say it’s noise when you look at the 10-year picture. You really need to stay focused on what you’re building and the vision.

Anything else you’d like to share?

My wife and I are expecting our first child this summer! It’s a boy and we’re very excited.

Check back soon for more posts in our Employee Spotlight series, and see if there is a role for you in our open positions here.