The rise of a Data Engineer
January, 4th of 2020: The beginning of a new intense journey.
Hello, my name is Pedro and this is my short story.
I am a 28 years old young man, born and raised in the suburbs of Lisbon, who went to college ten years ago thinking that the Marketing world would be my life. After some years of professional experience, a degree in Marketing, Advertising and PR and a master in Digital Marketing, I found that I was wrong.
After this short deception, and dealing with the difficulty of getting a new challenge on the Portuguese work market, I had to work about 5 years on the law market, not applying any of my scholarship knowledge, but trying to know more about this area until I felt I got stuck. Until one day.
Then, emerged the opportunity to join BiLD Analytics. A recent company on the market, with a strong scholar link and known by its effort to bring data analysis to a new level. And join how? — you ask. By applying the Big Data Academy: a two-month intensive training project about the whole Big Data ecosystem, from the concepts, to the tools and technologies.
First, I jumped. Then I thought about what I had just done.
It started just as a new professional challenge, for me. Then, by internalizing the whole change in my life, it became a challenge. Just it. A challenge, in all of the possible ways. The choice was made and I had to survive.
At the first days, it was all peaceful.
I had the opportunity to know BiLD’s HQ and all its staff, my colleagues — that now are my teammates, the tutors — that are my teammates either — and, last but not least, the trainers. Nothing scary.
In the following days, we took acknowledge of the next steps, the technologies we would learn and use, the career plan and all of those things that open the door to a new job or — as I may say — a new adventure.
Ok. It was something recent, very promising, completely outside my studying (and working) area and very challenging.
By joining BiLD Analytics (and the Big Data Academy) I “recovered” the opportunity to study again, to learn a world of new things (or close by) and challenge my limits when facing a steep learning curve. I couldn’t let it slip.
In this two-long-month intensive training, I went from a no-data-guy that thought that Clusters was just a cereal brand, to a kind-of-Data Engineer which can’t live without clusters to run his notebooks. And I didn’t do it all by myself.
The studying program was developed in order to help us to get through all the processes with the most natural ways. Or else, it was developed in a way that all the human resources with programming skills are united around the project, as we can “ping” them when we need either to clarify a doubt about the tasks to be developed, or to learn more about the professional life that will follow and in which they have more experience than us.
On this short-term learning path, I had the chance to learn how to use the best Microsoft tools. We had intensive training on SQL, python and PySpark in order to build ETL processes on Azure Databricks. We practised Azure Data Factory solutions. We used Visual Studio to create models. And, last but not least, used Power BI to display all the project results.
After the training period, I had the opportunity to integrate several full-remote projects, some in Portugal, some of them abroad. I’ve integrated projects from marketing and affiliates, to finance or sales and retail. I’ve worked with data transformation, reporting and visualization. Also, when I was out of project, I also had the possibility to obtain more than 20 certifications in tools in the area.
Most of it, after those two months — and all of the days after — I understood that we can always do more. After entering in the world of Data totally afraid and full of doubts, I think that I grow independent and my confidence grow either. My development skills went from “a guy who knew nothing” level to “an autonomous and accomplished professional” — with some technical gaps, of course.
This experience made me grow, professionally and personally. I think I needed this change to understand what my limits really are.
January, 21st of 2021: After a year of switching from certainty to risk, and known for the unknown, I cannot complain about the choice I made and the challenge it became.
The BigData Academy gave me the opportunity to know more about data and how can we manage it to produce a better decision. It also put some ultimate tools in my workday that I’ve never used — or imagined — before, which provided me with the chance to learn new technical skills and methods.
Now, I can tell you that risking on this was something that worth.
If you feel like the January 2020 version of me, stuck on your career, losing your time and delaying your dreams, think about this. Think about my story. It’s possible. If I did it, without any technical knowledge, you can do it either.
Try to list your challenges, fears and doubts. Now, list your dreams and your personal and professional accomplishments. Do they fit? If yes, ok. There’s no need to change. If not, you know what to do: Wait for a new edition of the Academy and enrol!
Today, if I can tell something about this whole adventure is that I’m thankful and glad to joined this BiLD’s experience because it represents a new era in my career and also a new opportunity to learn and grow.
Take your chance and never forget what Heraclitus told: “the only constant in life is change”.
PS: Now I can tell you that a Cluster is a group of resources that act like a single system and enable high availability and, in some cases, parallel processing. I mostly use them to run notebooks on Databricks, thing that you will know what it is after BiLD’s BigData Academy.
