Where are the jobs for Analytics Engineers?
A simple strategy to find the perfect position for you
While the focus on data analysis and AI continues to be strong, we feel that it is becoming increasingly difficult to find a position that matches what you want to do due to the development of data-related positions. In particular, for analytics engineers, which has been recently emerging, it is more difficult to find suitable positions and companies because of the following reasons:
- It is defined as part of the role of a data engineer or data analyst.
- In some cases, the position name is “Analytics Engineer,” but the job description indicates that it is not in the area that the job seeker expects to work in.
When looking for an analytics engineer position in a job search, it is an option to conduct a comprehensive search for data-related positions. However, there are certain characteristics of companies that tend to need analytics engineers, so to make the most of your limited time, you will want to narrow down your search as much as possible.
This is one of the areas that I struggled with during my most recent job search, so in this article, I would like to discuss the characteristics of companies in need of analytics engineers and how to search for them.
Role and Demand for Analytics Engineers
As a premise, the job description of an analytics engineer is, to quote the definition proposed by dbt Labs, an intermediate role between data engineer and data analyst.
The specific responsibilities are:
- Apply software engineering best practices such as DevOps to data transformation processes
- Ensure the quality of data provided to the business side, including data analysts
- Improve productivity of data analysis activities through documentation.
Some companies have further expanded the scope of their responsibilities which includes expanding the use of data within the organization and connecting to BI tools using a semantic layer.
Therefore, they must have both the engineering skills to manage and operate the data and the ability to effectively communicate with the business side that uses the data.
According to dbt Labs, it is not uncommon to see a tenfold increase in data analysis productivity by hiring an analytics engineer. If an organization of 50 people, each with 20% of their resources, is doing data analysis, and the hiring of one analytics engineer increases that productivity tenfold, that means that one person can generate the extra resources that amount to nine people, which is quite an astonishing achievement.
However, the number of jobs for analytics engineers is still small. Partly because they were born relatively young, job search on LinkedIn shows that there are still only about 4,500 jobs for analytics engineers worldwide, compared to 29,000 for data analysts and 27,000 for data engineers.
The reason is not that there are few demands, but rather that few companies have reached the stage where they need to create positions dedicated to analytics engineering. Even in companies that are in a phase where someone should be in charge of it, it seems that:
- some data analysts who have acquired engineering skills are doing it on their own
- there are job openings with a wide scope as data scientists, data engineers, or data architects that include the responsibility of analytics engineers
It is a reasonable strategy for job seekers to start with the list of positions called “Analytics Engineer” but this alone will limit the number of positions for which they can apply. Therefore, it is a good idea to look at other data-related positions and determine whether there is an urgent need for analytics engineering from the perspective of industry and culture.
Industry
Needless to say, Companies that need an analytics engineer are those that already understand the value of the data they possess and have reached a certain stage of utilization, such as having a competent data organization and a high need for data maintenance.
Only a limited number of companies have reached this stage, and by utilizing their data through data analysis and developing machine learning models, companies that can further enhance their products, such as web services and mobile applications, are ahead of other companies. Another reason may be that data organizations can exist close to the product side, making it easier to secure resources and collaborate.
In particular, companies that provide mobile apps and AI-based analysis for data generated in non-IT industries often need analytics engineers. For these companies, data is not just for marketing analysis or business management, but also it is the product itself, so making the most of the data they have directly results in the success of their business.
In addition, these companies are replacing the digital transformation projects that existing businesses have been unable to achieve. When tech companies receive raw data from other companies, process it, and offer it as a product, or when they have to deal with data of unstable quality such as text data, it is highly difficult to guarantee data quality that meets the standard for a product. In other cases, where the quality of data provided is life-threatening, such as in the medical field, or areas with strict legal restrictions, the importance of quality assurance increases due to the business risk posed by low-quality data. As a result, the demand for analytics engineers who specialize in these areas is growing.
As an opposite example, in the management of an expensive restaurant, data can be used for marketing activities and inventory control, but there are many issues to which data cannot contribute much, such as how to prepare high-quality ingredients and how to hire excellent chefs. In these industries, unless there is a top-down consensus to allocate resources to the IT and data organizations, or unless executives give the data team independent authority in the form of spin-offs, etc., there will be competition for resources within the organization. Therefore, cultural support is vital.
Culture
Non-tech companies, as in the previous example, face a higher level of difficulty in promoting data utilization. However, if the company has succeeded in fostering a culture in which data utilization pervades the entire organization, with the support of highly motivated executives, there will be cases where analytics engineers will be needed.
Some of the companies that train their business-side employees to have a certain level of data analysis capability, so that data analysis using BI tools and SQL is conducted in their daily work.
Other companies have affluent resources of data analysts, not only in the central data organization but also in each business department. In these companies, there is a lot of data analysis and decision-making taking place daily in each of the teams. While this in itself is desirable, the data analysis by each team without any governance can lead to problems such as data quality not being guaranteed and data being defined in different ways.
For instance, the management team may receive reports from Team A and Team B, but the numbers are slightly different, and it may be difficult to determine which is correct. Or worse, each team may make the opposite decision based on the inaccurate data.
In such a situation, there is an incentive to create a new team that can control the myriad of data analyses taking place in various teams and guarantee a certain level of quality. The new team can then take on the roles of creating a DWH from raw data that is easy to use for analysis, unifying data definitions, and assuring data quality, thereby reducing the workload of data analysts and increasing their productivity.
Although we have not yet seen many companies (especially in Japan) that have been able to achieve advanced data utilization in non-tech companies, the number of companies promoting the use of data is likely to increase in the future, encouraged by the AI boom.
A better way to find analytics engineer jobs
Companies that want to use data begin by hiring data analysts and data scientists for business positions as a first step, but as data use expands to a certain scale, they need analytics engineers to guarantee data quality to improve operational efficiency.
To find analytics engineer jobs, it is a better strategy to approach companies that not only have a job opening as an “Analytics Engineer” but also publish its advanced use of data to attract candidates. You can work on it at these companies as one of the data-related jobs even though there is no position called “Analytics Engineer”.
If there is a job called Analytics Engineer, the company at least knows the name of the relatively new domain, and it also indicates that managers have a deep understanding of trends in data utilization and have already achieved a certain level of success.
If you want to know whether you can work in the area of analytics engineering from other data expert positions, in addition to reading the job description, you can ask the following questions to get some additional information as to whether the companies have reached the phase where they need an analytics engineer.
About the size and structure of the data team
- Has the company successfully hired a minimum of data scientists and data analysts as an initial phase?
- Does the company deal with data engineering tasks by hiring a full-time data engineer or introducing managed services that can be operated by business staff?
Data use by the business side
- Are there a large number of BI tool users, and are they able to use the tools even by non-data analysis staff such as Sales?
- Are there a certain number of business people who can write SQL and retrieve data from DWH by themselves?
As for the long-term trend, the demand for analytics engineers will correlate with the demand for data analysts and data scientists, since analytics engineers cannot exist without the users of well-prepared data. Just as the classical data engineering domain is being replaced by managed cloud services, analytics engineers may also become unnecessary as the demand for data analysis jobs disappears due to some innovation. Nevertheless, generative AI, which is currently seen as a threat to many jobs, seems an ally for analytics engineers in the foreseeable future by automating tasks such as documentation and repetitive SQL query creation.
