Data Engineering 101
Is Data-Engineering a Great career move in 2022?
Data Engineering is the new Sexiest Job of the century
In October 2012, HBR predicted that Data Science will be the sexiest job in the 21st century.
For the first 10 years of the century it did seem to be exactly matching.
The highly quoted report was perfect to predict a huge surge in demand for Data Scientists as resources. This also gave rise of plenty new startups ( coursera / Udemy etc.) that specialize in training the corporate employees on relevant skills.
Don’t believe me? Read the article below…
However, post 2019 in the Pandaemic era, things have completely changed. After seeing a record 80% growth YOY in 2019, Data Science role for recruitment actually grew by only 10% in 2020. You ask why?
The answer is growth of Data Engineering.
Data Engineering in comparison is growing 40% YOY as per the study published in interviewquery.com below…
The demand for Azure Data Engineer especially is astronomically high in the current Indian job market. As per this Dice report Data Engineering grew 50% YOY in 2020.
However you might still ask, Is it future proof?
Ohh Yes, Absolutely.
- Ideally data should answer this question automatically. And as long as Data is explored to answer questions in the business, Data Engineering demand will remain in one form or the other.
If Data Engineering basics and fundamental logics are clear, you are ready to work on any new shiny tool in the Data space.
Due to a lot of growing complexity of different kind of sources to get the data from, more and more enterprises are trying to gather as much data as possible. This again increases the complexity of the Data Pipelines and driving demand for Data Engineers.
Ok, so there is demand for the role, but can money match the demand?
2. So, How much does a data Engineer get paid?
As per Glassdoor, Ambitionbox and Payscale the average Data Engineer Salary in India is 8–9lakhs per annum.
Just to put that into perspective, the salary is dependent on your work experience and the company you are working for.
However the salary can range from 3–4lakhs for freshers to upwards of 30lakhs for people with 10+ years of experience.
Don’t believe me? Look at the chart below to understand correlation with experience.
Check the detailed report here….
The average salary is on per with Software Engineers or even more at times.
However looking at linkedIn one must have noticed there is high demand from cloud data engineers. With increased demand you can expect higher returns and easier to crack an interview.
So, the perpetual question then, what’s better?
3. Data Engineering or Data Science……?
Demand for Data Scientist is slowing down, as enterprises have understood that without quality data no Data Scientist or DS algorithm can provide the results expected.
FAANG companies and other big-tech are reaping real benefits of Data Science and ML as they have heavily invested in building the back bone Data Infrastructure. But in most other companies, the Data maturity is still not up to the mark, and hence the buzz around DS is finally slowing down and that is great news for Data Engineers.
Cracking a DS interview is also much harder in comparison to DE interview.
DS learning curve is also much longer as it involves complex knowledge of Maths, Algorithm and Statistics too other than programming and Data tools.
But you have good news, for DE all you need to focus on is understanding of Data and related tools and little programming.
So, how much is the demand for Data Engineering and why?
4. High Demand:
The lock-downs and restrictions post COVID 19 have expediated on-prem to cloud migration for many enterprises.
This has resulted in huge spending for moving data and operations to cloud. Even though Big Data and cloud have remained a buzz word since 2015, they actually came into reality in 2020–21, where large scale transformations took place.
As a result at the moment there is huge demand for Cloud Data Engineers and Big Data Engineers to help enterprises move from on-prem to cloud and build a scalable Data infrastructure.
The demand is here to stay at least for the next 2–3 years, which will mean a lot of easy opportunities to crack the interviews.
Too good to be true?
According to DICE’s 2020 tech job report Data Engineering was the fastest growing tech job beating Data Science hands down, the demand has only increased since 2020.
So, can you create a lot of wealth becoming Data Engineers?
It’s been believed for a long time that “Data is the new Oil” , however when Oil was found a new set of people also became filthy rich, people who had the skills to extract Oil.
When people went digging for Gold, the person who become the richest was the one who sold the shovel.
So, if Data is the new Currency / Gold / Oil the people who know how to store, process and get intelligence from the Data will become extremely successful too.
Data Engineers are those people and with the right skills and tools 2022 is going to be a prime year for Data Engineers.
Now that we have covered the Why? Let us discuss the How?
The natural question once you have understood the demand, will be How do I become a Data Engineer?
I have provided a detailed roadmap on the same. This involves comprehensive list of topics to cover, and even detailed sub sections for learning in depth. The roadmap also has a learning plan (a notion template) that you can follow along and mark your progress.
I will be sharing more stories, writings, and experiences in the data industry. You can follow me for more posts like this.
Thanks for reading! If you want to get in touch with me, feel free to reach me at withsaikat@gmail.com or my LinkedIn Profile.
