Creating Opportunities with AI at Scale
Interview with Deepak Agrawal, LinkedIn VP Engineering & Head of AI
Deepak Agarwal is Vice President of engineering at LinkedIn where he is responsible for all AI efforts across the company. Well known for his work on recommender systems he has published extensively in top-tier computer science conferences, has co-authored several patents, is a Fellow of the American Statistical Association, and is also an associate editor of two flagship statistics journals.
Having worked previously at AT&T and Yahoo and served on the Board of Directors at SIGKDD, Deepak is no stranger to the corporate world. Expert in Artificial Intelligence technologies and engineering leadership with more than twenty years of experience developing and deploying state-of-the-art machine learning and statistical methods for improving recommender systems, we can’t wait for him to tell us more at AI With the Best online developer conference 29–30th April! For now we’re thrilled to have asked him some of our questions.
Q How does LinkedIn use machine learning across products?
At LinkedIn, machine learning is like oxygen, it powers every single experience for our members. We use machine learning to harness the wealth of information in our identity and network assets to help connect talent with opportunity at scale. Machine learning is used to recommend people you can connect to, recommend information about your network and professional news in the newsfeed, recommend jobs you can apply to enhance your career, courses you can take to enhance and broaden your skills, advertising, consumer and enterprise search, and many others.
Q What are the risks involved in trusting a machine to display recommendations to over 460 million users — are there sometimes mistakes in the models?
ML algorithms are statistical in nature and hence prone to making mistakes. For instance, we have instances where a member is recommended a job for which she is overqualified. We consistently work to teach algorithms to reduce errors by learning from past mistakes. Every single ML scientist is constantly thinking of ways to improve the performance of algorithms by building new models and adding/deleting components of existing ones. It is also important to quantify the cost of various kinds of errors and provide that as input to the algorithms. Despite the errors made by algorithms, it is the only feasible way to scale the system in a cost-effective fashion to provide value to our members. The benefits of machine learning to display recommendations far outweigh the risks we take with sporadic mistakes.
Q What do you personally find most exciting about your current role?
Using data to make decisions in the face of uncertainty is something that has always fascinated me since high school. To use this technology and change the world by connecting talent with opportunity at scale and making a difference in the lives of so many people on the planet is the most exciting part of my current role. I am also very fortunate to be leading a stellar team of folks who are so passionate about what they do. It is a joy and a humbling experience to be working alongside so many smart people every single day and learning new things.
Q Tell us about your latest research project/latest release?
We recently released salary insights for professionals on LinkedIn. It was a fascinating project since it involved making inference about cohort salaries without having access to data at member level to preserve privacy. We are a member first company and take member trust and privacy very seriously. Doing statistical inference while preserving privacy was quite challenging and fun. I also liked the fact that we had to estimate confidence intervals. This is more challenging than obtaining point estimates.
QCan you share some insights into LinkedIns plans for AI over the next few years?
We are constantly expanding our AI efforts and investing in new cutting edge technologies like deep learning, chatbots, knowledge base, machine reading, privacy preserving machine learning, experimental design with network interference, and many others. We believe AI is an integral component of all our existing and future products. We will continue to innovate in this space to connect professional opportunities at scale and make the world a better place. Our goal is to use AI to create opportunities for the human race.
Q How do you ensure collaboration and smooth organisation in your team?
We only hire smart folks who are passionate about our mission and are a strong culture fit. The vision and mission of LinkedIn is inspiring. Every team member knows they are part of something that is much bigger than themselves. But they also realize the audacity of the goal and know it cannot be achieved by working in silos. This fosters collaboration across the board. You can think of me as the gatekeeper of the vision and mission, I keep reiterating it.
I also spend a lot of time keeping the organization of efforts as unambiguous as possible among teams and ensuring accountability across the board. We have a strong culture of demanding excellence. We encourage folks to take bold risks but to fail fast. We are obsessed with rewarding innovation, we do not celebrate small incremental progress. The focus is very applied ML: it is always about building great products, not building great models.
Q What advice would you give to budding AI developers?
Spend a lot of time understanding your domain and framing the right problem to solve. Tools to bring AI to your products is becoming a commodity, the last mile problem is the most difficult part for AI. Keep an eye on privacy and model interpretability, it is going to become very important in the future (you cannot escape it). If you are working on problems that requires constant updating of models on a regular basis, you should pay attention to reliability. It is better to take a small hit in accuracy to achieve higher reliability for your workflows. Finally, read a lot and keep experimenting with new ideas.
Q Are you excited about speaking at AI With The Best ? What made you want to be a speaker?
I am thrilled to be speaking at AI With the Best. It is an honor to be part of such a talented team of speakers. AI is going to be like electricity in the next few years, efforts like these are important to both inspire and train the future workforce in this area.
Thank you Deepak!
You can ask Deepak your own questions, and learn more in his talk about using AI for creating Professional Opportunities at scale at our upcoming AI With The Best, Online Developer Conference 29–30th April.
