From Novice to Researcher; Two years of deep learning with Fast.ai

Upon returning from paternity leave this March I’m excited to share that I’ve joined NVidia to do applied deep learning research full time, working at the intersection between Tabular (sometimes called structured) data and rapids.ai. This journey wouldn’t have been possible without the amazing course, library, and community that is fast.ai.

Like Sylvain Gugger, another alum of the course who is currently working as a Research Scientist for fastai and whose blog post inspired this one, fastai has changed my life and I’m deeply grateful to Jeremy and Rachel for their vision of a more accessible deep learning education for all. In the spirit of his post I wanted to share my journey and the things that have helped me along the way in the hopes they will aid and inspire others who are starting down this path.

The Journey

Unlike many fastai students I come from a machine learning background. One more focused on human computer interaction, and pre deep learning, so the work I did over the course of my thesis could be implemented by a fastai student in hours, but the foundation of math/programming and the ability to read papers definitely helped. Still, I rarely refer back to that time and I agree completely with Jeremy’s assessment that a PhD isn’t required to do deep learning work.

Following University I worked for the online dating site plenty of fish, writing matching algorithms, fraud detection systems, and eventually leading the research team. I left pof the year following it’s sale to the Match group, and found myself at a crossroads, looking into data engineering options and not really interested in returning to data science.

The Fastai course v1 came out that same spring, and as I worked my way through it and got to know people on the forum I got more and more hooked. Top down, applied development is a learning approach that works well for me, and I quickly fell in love with Jeremy’s teaching style. When I found out there was a part 2 I applied immediately and literally jumped for joy when I found out I’d gotten in.

With my background in recommender systems, I was very interested in that component of the course, and began researching into that area more and more. In the process I landed a role at Realtor.com working on recommendation and search ranking projects, and it was there that I really began to see the potential of what I was being taught. Deep learning for recommender systems became my passion, and I dove in head first, reading every paper on the topic, and consuming everything I could find on Youtube. I’ll talk more about the habits I’ve formed over the last two years that led to the research role in the following section, but this time was very formative for me in terms of developing the skillset and knowledge base that led to my current role at NVidia.

Along the way I’ve taken every single offering of fastai. I’m currently on my third iteration, and every class I learn something new. If you haven’t watched the old courses I highly recommend going back, and the same thing goes for the machine learning course, which taught me what I know about random forest models. The quality of education provided by fastai is so high I structured my team around it at Realtor, using classes as jumping off points for projects, which proved extremely successful.

The steps along the way

While my journey is unique to me, I want to offer some help to those who want to achieve the same dream of becoming a deep learning researcher or a better practitioner.

Tips for how to effectively complete fastai abound on the forum, and I highly recommend you look them up and follow them if you aren’t already. Going beyond the class takes additional effort, but in my mind it’s when the magic really starts to happen. A number of habits have really helped me become a better researcher and a better developer. They’re simple, but effective. It’s easy to breeze through them like you can shift-tab through the notebooks, but if you can implement them I guarantee they’ll have an impact.

1. Get comfortable with papers. Check Arxiv weekly, read a paper every night. I used to use arxiv-sanity but sadly it doesn’t seem to be maintained anymore. I’m still looking for a solution so if you have something let me know. I would recommend picking a specific research area where you can read most of the papers coming out. Find a topic that interests you like recommendation or style transfer and keep up on that small field. Getting in the habit of regularly reading papers is now my super power. It’s one you could have too if you form the habit.
2. Convert the fastai videos to audio only to listen to as a ‘podcast’ on the go. For me finding time to watch videos twice or three times isn’t feasible, particularly now that I have two young boys, but the density of the material requires it. Watch it once, and listen again and again on your commute, walk or wherever. I’ve listened dozens of times and I still pick up new concepts and ideas.
3. Implement something that’s different than the notebooks provide, end to end. This was the key for me to unlock the potential of fastai. I recreated the denoising auto encoder from the solution to the porto seguro safe driver competition. That turned out to be the starting point for a whole range of research into deep learning on tabular data and led to my current role. Pick something different from what most people are implementing, and implement it well.
4. Stay active on the forums and participate in the community. It’s one of the most overlooked aspects of the course in my mind, offering a whole bunch of people just as excited about deep learning as you are. For me it’s been a big catalyst of ideas, knowledge, and interest.
5. Look for other courses to expand your horizons. I highly recommend Stanford’s NLP course CS224n taught by Chris Manning, CS231n taught by Andrej Karpathy, Gene Kogan’s course on deep learning applied to art The Neural Aesthetic, and PyData which regularly features great deep learning talks.

Fastai has given me so much, and continues to do so, and I’m excited to be moving into a position where I can finally give something back. If you’d told me at the beginning of my journey when I saw the first video on youtube that a free online MOOC would lead me to a research role at one of the top tech companies in the world I’m not sure I would have believed you, but here I am. I have no doubt that it’s almost entirely due to fast.ai.