How I went from Automation Engineer to Data Scientist
Advice and lessons learned from my journey into the world of Data Science and Machine Learning.
When I joined PayPal as a Software Automation Engineer, I never imagined that 3 years later I would be working as a Data Scientist and Security Researcher. Surreal as it sounds, that is my reality now.
In this blog post, I would like to share some insights I learned throughout this journey in hopes that it will help any reader who is contemplating making a career transition. For me, that transition was from Software Engineer to Data Scientist, but I believe that most of these insights apply to any kind of career change.
So here it goes…
First, find your passion!
“Pursue your passion, and everything else will fall into place. This is not being romantic. This is the highest order of pragmatism.” ~ Gabby Giffords
I didn’t always know I wanted to be a Data Scientist. In fact, I didn’t give any thought to career planning before I started working at PayPal. What helped me a lot in that sense was a mentor from within the company that encouraged me to ask myself the right questions and pushed me to take charge of my career path.
So I guess my first tip is to take time and think about things like:
What motivates you? What aspects of your work do you enjoy most? What aspects do you least enjoy? Where do you see yourself in 1/3/5 years?
If you can find a mentor to accompany you during this self-discovery process, even better.
Baby Steps
“It is better to take many small steps in the right direction than to make a great leap forward only to stumble backward” ~ Old Chinese Proverb
OK, let’s assume that this discovery process brought you to the realization that you want to make a career change.
The next thing you need to understand is that this type of change doesn’t happen overnight. It’s a process that is comprised of many small steps. In my case, the transition took more than 2 years.
Let’s start at the beginning (best place to start :)). I was first exposed to the world of Data Science and Machine Learning when I started working as an Automation Engineer at PayPal’s Security Product Center. We had a really strong Machine Learning team that just started conducting weekly reading group sessions. In each session, one of the team members would present a paper that is either relevant to his current project or just interesting to them personally and cultivate a discussion.
At first, I attended these sessions in “listen-only” mode. From session to session I slowly started to pick up more and more concepts. After a few months, I was requested to present a paper myself. I found a paper that I felt was interesting and showed it to the ML team lead David. Since the authors of the paper published their code, David suggested that I should try to replicate their results. I quickly learned that this wasn’t such an easy task as the code was incomplete and not very well documented, but this was a great learning experience for me. It was my very first experience with machine learning code and I got a chance to understand how a lot of basic concepts (for example, backpropagation) are implemented. Another important lesson that this experience taught me, is the importance of reproducibility in academic research and that just because something is written in a paper doesn’t automatically mean that it is true.
The next small step I took was volunteering to give an internal introductory talk on Reinforcement Learning (this helped). Not because I knew anything about Reinforcement Learning at the time, I just found it to be very interesting (AlphaGo Zero was released around the same time). I believe that there is no better way to learn than to teach, so this was a perfect opportunity for me to learn about the topic.
Be persistent!
“Success is the sum of small efforts, repeated day-in and day-out.” ~ Louis Sachar
At this point, I already realized that this is the path I wanted to pursue, so I started to go into high gear in terms of my self-learning. This included taking Andrew Ng’s amazing Machine Learning course followed by deeplearning.ai’s Deep Learning specialization, reading blog posts and papers, following influential researchers and practitioners on Twitter and LinkedIn, subscribing to Machine Learning and Deep Learning DL’s, attending relevant meetups and even listening to ML podcasts while cleaning my house.
I was determined to do everything in my power to put myself in a position in which when an opportunity presented itself, I would be ready for it. Every night, once my kids went to sleep, I would open my laptop and study (special thanks to my amazing wife who put up with me during this time :)). That became my routine for about 2 years.
But this isn’t enough. To make any career change, it’s best to have the support of your managers. Which brings me to my next point.
Be explicit regarding your career aspirations!
“The world will conspire with you in your career journey, if you ask” ~ Sri Shivananda (PayPal’s CTO)
I am fortunate to be working for a company that highly encourages personal growth and personal development. This is not something to be taken for granted. Even if the same applies to you, don’t assume that people around you will guess what your ambitions are and that things will magically happen by themselves. You have to give people the opportunity to help you. Once people around you are aware of your aspirations, opportunities are likely to start presenting themselves.
Embrace every opportunity!
Especially if you feel that it’s a challenge and you’re not yet ready for it. These are exactly the experiences that will help you grow the most.
“If somebody offers you an amazing opportunity but you are not sure you can do it, say yes — then learn how to do it later!” ~ Richard Branson
I was lucky enough to have the support of my managers as well as the encouragement and guidance of the Machine Learning team lead at the time, which led me to participate in several projects with the team on top of my daily work as an Automation Engineer. For example, I was given the chance to participate in a research project as part of the collaboration between BGU and the PayPal Security Product Center. The project was in the domain of AI Planning. I got to contribute to the empirical experiments and even co-author a paper.
Another meaningful experience I had during that time is mentoring a group of 4 Israel Tech Challenge fellows during their 5 weeks internship project phase of the program. The project aimed to use Machine learning to statically analyze network traffic and determine the origin of the traffic (physical machine a virtual machine or a Docker container). We got pretty good results and even submitted a patent based on the idea.
There is no chance I would have gotten these amazing opportunities if I hadn’t let my managers know of my ambitions!
Get your hands dirty
“Knowing is not enough; we must apply. Willing is not enough; we must do.” ~ Johann Wolfgang von Goethe
Once you feel you have the basics down, you should get as much hands-on experience as possible. True expertise is achieved through practice. Every time you learn a new concept, try to actually “get your hands dirty” and play around with some code. Kaggle is a great platform for getting hands-on experience, and I highly recommend it. For those of you who don’t know it, Kaggle is a very active online community of data scientists and machine learners, owned by Google. The community is very supportive of newcomers to the field.
A tip that can go a long way in terms of getting practical experience is finding a partner in crime — someone who is equally (or at least somewhat) passionate about the subject that can accompany you through your learning journey. Once you decide on learning goals or milestones together, you are both less likely to cut corners or decide to skip the practical aspect of the learning experience. It also always helps to have someone to bounce ideas off of or consult with in case you are struggling with a problem.
For me, that person was a co-worker and friend that was already a data-scientist and security researcher in a different team. We started participating in Kaggle competitions after work hours.
At first, we tackled past competitions and after a few months transitioned to live competitions. We managed to get 3 medals (2 Bronze and 1 Silver) out of the first 5 live competitions we entered, which goes to show that you shouldn’t be afraid to jump in and start practicing your newly learned skills. The nice thing about Kaggle is that there is a variety of competitions in various domains which means that you can always find a competition that is interesting for you or relevant to your current work so that you can utilize domain expertise that can help you put yourself ahead of the pack.
Don’t wait for the perfect moment!
“If we wait until we’re ready, we’ll be waiting for the rest of our lives.” ~ Daniel Handler, The Ersatz Elevator
There will always be more to learn. Don’t wait for a moment in which you feel you are ready to make the transition, because that moment might never come. There will always be new algorithms to learn, new MOOCs you can take, new libraries to know, more skills to acquire. We live in an era in which technologies are constantly changing and evolving, natural curiosity and the ability to self-learn are more important than any specific skill.
For me, what helped me come to this realization is my mentor who simply asked me during one of our sessions: “What are you waiting for? What’s keeping you from applying to Data Science positions?” He was right. At that point, I was already learning intensively for about 2 years and had sufficient hands-on experience to rely on in any technical interview. I decided to take his advice and start interviewing internally (with the support of my manager) to data science positions, which led to me joining the Threat Sciences team as a Data Scientist and Security Researcher.
A few closing notes
- This advice is based on my personal experience. These are the things that worked for me. There is no right or wrong way when it comes to career transitions and I believe that every person should try and find their own path.
- There is a common conception that when you make a significant career change, you are basically starting from scratch. I don’t believe that this is true. You might be entering a new field, but that doesn’t mean that all of your past experience becomes irrelevant. My Software Engineering background has made me proficient in things like writing modular and reusable code, version control and continues integration. My Software Automation experience also greatly contributes to my day to day work and provides me with a unique perspective that a “traditionally trained” data scientist might be missing. For example, I am very passionate about applying the same quality standards that are practically inherent in “traditional” Software Development into the world of Machine Learning (Unit Testing, Data Validation, etc..). I even gave a talk about it at PyconIL. All that to say that you should be aware of your strengths and the unique perspective and capabilities that you bring to the table. A diverse team is a stronger team!
- PayPal allows for amazing learning opportunities and resources to grow. You should come work with us.
That is what I wanted to share, I hope some of you will be able to apply these learnings into your personal career journey. Good Luck!
